# A/B variants

> Test up to four versions of your agent's first message, pause or keep a variant, and read what each version sent and how many replies it got.

Source: https://docs.whatsetter.com/dashboard/ab-variants/

An A/B test sends different versions of your agent's first message to
different leads, so you can keep the one that works best. A test holds up to
four variants: A, B, C and D.

## Where the A/B test is now

Open **Agents**, click your agent, open the **Prompt** tab, then the
**First message** step. The A/B test lives on that step. It is no longer in a
campaign panel: campaigns are now called agents, and the old Prompts page is
now the **Prompt** tab of each agent.

Above the text, click **More options** (the three dots), then
**Test a 2nd version (A/B)**. Once a test exists, a button with the name of
the variant you are editing (**Variant A** by default) appears on the left of
that bar. It lists the variants and opens **Compare variants** and
**Manage the A/B test…**.

Creating, pausing or deleting a variant needs the
**Edit agents and their prompts** permission. See
[Team](/dashboard/team/).

## Create a second version of the first message

1. Open **Agents**, click your agent, open the **Prompt** tab, then the **First message** step.
2. Click **More options** (the three dots above the text), then **Test a 2nd version (A/B)**. A window opens.
3. In that window, click **Test a 2nd version (A/B)** again. Variant B is created as a copy of the text in your editor, and the editor switches to it.
4. Change one thing in variant B, then click **Publish**.

Variant B joins the test the moment it is created, with the text it was
copied from. A draft is never sent to leads: a change reaches them only when
you click **Publish**.

To add a third or a fourth variant, click the variant button, then
**Test a version C**. A test stops at four variants: A, B, C and D.

## How new leads are split between variants

Open **Agents**, click your agent, open the **Prompt** tab and the
**First message** step, click the variant button, then
**Manage the A/B test…**. Each row shows the share of its variant, for example
**In test · 50%**.

- Active variants share new leads equally: 50% each with two variants, 33% with three, 25% with four. There is no setting for another split such as 70/30.
- One lead receives one variant. The variant is recorded on the lead and on its first message.
- For leads coming from a form or the API, the next first message goes to the active variant whose current version has sent the fewest. A variant you add or publish again later therefore gets the next leads until it has caught up.
- For a list, each contact is given its variant from its phone number, so the shares even out over the whole list.

## Pause a variant or put it back in the test

Open **Agents**, click your agent, open the **Prompt** tab and the
**First message** step, click the variant button, then
**Manage the A/B test…**. Each row has a switch.

- Switch off: the row reads **Paused**. The variant is not sent to new leads any more, and the other variants share them.
- Switch on: the variant returns to the test with an equal share.

Leads who already received a paused variant keep their conversation. When one
variant only is active, every new lead gets it. When no variant is active,
your agent uses the last first message you published.

Publishing a paused variant puts it back in the test: pause it again if you
only wanted to update its text.

## Compare the variants

Open **Agents**, click your agent, open the **Prompt** tab and the
**First message** step, then click the variant button.

- **Compare variants** shows the variants side by side: an excerpt of each text, its share of new leads and its numbers.
- **Manage the A/B test…** shows one row per variant with what it sent, how many leads replied and the reply rate. A variant that has sent nothing reads **Awaiting data**.

Above the rows, one line names the variant that is ahead, or reads
**No clear winner yet**. Its label compares the two variants with the best
reply rates:

| Label | When it shows |
|---|---|
| **too early** | One of the two has fewer than 30 sends, or the gap is under 1 point. |
| **trend forming** | Both have at least 30 sends and the gap is 1 point or more. |
| **high confidence** | Both have at least 80 sends and the gap is 4 points or more. |

The numbers belong to the current version of each variant. Saving or
publishing a new version of a variant starts its numbers again at zero.

## Keep the best variant and stop the test

1. Open **Agents**, click your agent, open the **Prompt** tab and the **First message** step.
2. Click the variant button, then **Manage the A/B test…**.
3. Click the row of the variant you want to keep, then **Keep variant B** (the button carries the letter of the selected row).
4. Confirm.

The variant you keep is sent to 100% of new leads and the test stops. Its row
reads **Winner: variant B**. The other variants are not deleted: they are
paused. If the numbers put another variant ahead, the confirmation window
warns you before you confirm.

To start the test again, select the winning row and click
**Resume the test**: every variant returns to the test with an equal share.

## Rename or delete a variant

Open **Agents**, click your agent, open the **Prompt** tab and the
**First message** step, click the variant button, then
**Manage the A/B test…**.

To rename a variant, click the pencil next to its name (**Click to rename**),
type up to 80 characters and press Enter. An empty name brings the letter
back, for example **Variant B**. Publishing replaces that name with the
**Version name** of the publish window: type your name there to keep it. If
you keep the date it suggests, the variant shows its letter again.

To delete a variant, click its row, then **Delete this variant** and confirm
with **Delete the variant**. The link shows only when the test has at least
two variants.

> **A deleted variant does not come back**
The variant is archived with its whole history and its numbers. The dashboard
has no button to restore it. Leads already in conversation keep their thread.

## The A/B test of the Conversation step

Open **Agents**, click your agent, open the **Prompt** tab, then the
**Conversation** step. **More options** offers the same
**Test a 2nd version (A/B)** entry, with the same window, rows and switches.

- A lead is given a variant when the agent first answers it, and keeps that variant for the whole conversation. If its variant is paused or deleted, the lead gets another one.
- The window adds **Qualification after a reply**: of the leads who replied under each variant, how many became qualified.
- Each variant can book meetings or not. Open the **Settings** tab, then **Meetings**: the block **By behavior variant** has one switch per variant. See [Meetings](/dashboard/bookings/).

## See which variant a lead received

- In **Conversations**, a message written by your agent under a variant carries its letter after the agent's name, for example `· variant B`. Hover it to read the variant's name. When one agent is selected and its conversation test runs, the filters also offer **A/B variant**.
- On the agent's **Results** tab, a line tells you that a test is running, with the link **See the test**.
- In **Analytics**, under **Compare**, the agent's line reads **A/B test running** and opens its **Prompt** tab.

Analytics no longer has an A/B scoreboard: the numbers of a test are read in
the **Prompt** tab, in the window opened by **Manage the A/B test…**.

## Why a variant gets no new leads

Open **Agents**, click your agent, open the **Prompt** tab and the
**First message** step, click the variant button, then
**Manage the A/B test…**, and check the row of the variant.

- It reads **Paused**: turn its switch on.
- Another row reads **Winner: variant A**: the test was stopped. Select that row and click **Resume the test**.
- You changed its text without publishing: a draft is never sent. Open the variant and click **Publish**.
- It has sent more than the others: for leads coming from a form or the API, the variant that has sent the fewest gets the next leads first, so the one ahead waits.
- No first message leaves at all: the agent is paused, its number is disconnected, or no new lead is coming in. See [Agents](/dashboard/agents/) and [Lists & import](/dashboard/lists/).

> **A clean test changes one thing**
Change one thing between two variants: the hook, the closing question or the
length. Then leave the variants alone until each has enough sends, because
every new version of a variant restarts its numbers.

## Related

- [Your agent's prompt](/dashboard/prompt/)
- [Test your agent](/dashboard/playground/)
- [Meetings](/dashboard/bookings/)
- [Analytics](/dashboard/analytics/)
