Your agents
A/B variants
Test up to four versions of your agent's first message, pause or keep a variant, and read what each version sent and how many replies it got.
An A/B test sends different versions of your agent’s first message to different leads, so you can keep the one that works best. A test holds up to four variants: A, B, C and D.
Where the A/B test is now
Section titled “Where the A/B test is now”Open Agents, click your agent, open the Prompt tab, then the First message step. The A/B test lives on that step. It is no longer in a campaign panel: campaigns are now called agents, and the old Prompts page is now the Prompt tab of each agent.
Above the text, click More options (the three dots), then Test a 2nd version (A/B). Once a test exists, a button with the name of the variant you are editing (Variant A by default) appears on the left of that bar. It lists the variants and opens Compare variants and Manage the A/B test….
Creating, pausing or deleting a variant needs the Edit agents and their prompts permission. See Team.
Create a second version of the first message
Section titled “Create a second version of the first message”- Open Agents, click your agent, open the Prompt tab, then the First message step.
- Click More options (the three dots above the text), then Test a 2nd version (A/B). A window opens.
- In that window, click Test a 2nd version (A/B) again. Variant B is created as a copy of the text in your editor, and the editor switches to it.
- Change one thing in variant B, then click Publish.
Variant B joins the test the moment it is created, with the text it was copied from. A draft is never sent to leads: a change reaches them only when you click Publish.
To add a third or a fourth variant, click the variant button, then Test a version C. A test stops at four variants: A, B, C and D.
How new leads are split between variants
Section titled “How new leads are split between variants”Open Agents, click your agent, open the Prompt tab and the First message step, click the variant button, then Manage the A/B test…. Each row shows the share of its variant, for example In test · 50%.
- Active variants share new leads equally: 50% each with two variants, 33% with three, 25% with four. There is no setting for another split such as 70/30.
- One lead receives one variant. The variant is recorded on the lead and on its first message.
- For leads coming from a form or the API, the next first message goes to the active variant whose current version has sent the fewest. A variant you add or publish again later therefore gets the next leads until it has caught up.
- For a list, each contact is given its variant from its phone number, so the shares even out over the whole list.
Pause a variant or put it back in the test
Section titled “Pause a variant or put it back in the test”Open Agents, click your agent, open the Prompt tab and the First message step, click the variant button, then Manage the A/B test…. Each row has a switch.
- Switch off: the row reads Paused. The variant is not sent to new leads any more, and the other variants share them.
- Switch on: the variant returns to the test with an equal share.
Leads who already received a paused variant keep their conversation. When one variant only is active, every new lead gets it. When no variant is active, your agent uses the last first message you published.
Publishing a paused variant puts it back in the test: pause it again if you only wanted to update its text.
Compare the variants
Section titled “Compare the variants”Open Agents, click your agent, open the Prompt tab and the First message step, then click the variant button.
- Compare variants shows the variants side by side: an excerpt of each text, its share of new leads and its numbers.
- Manage the A/B test… shows one row per variant with what it sent, how many leads replied and the reply rate. A variant that has sent nothing reads Awaiting data.
Above the rows, one line names the variant that is ahead, or reads No clear winner yet. Its label compares the two variants with the best reply rates:
| Label | When it shows |
|---|---|
| too early | One of the two has fewer than 30 sends, or the gap is under 1 point. |
| trend forming | Both have at least 30 sends and the gap is 1 point or more. |
| high confidence | Both have at least 80 sends and the gap is 4 points or more. |
The numbers belong to the current version of each variant. Saving or publishing a new version of a variant starts its numbers again at zero.
Keep the best variant and stop the test
Section titled “Keep the best variant and stop the test”- Open Agents, click your agent, open the Prompt tab and the First message step.
- Click the variant button, then Manage the A/B test….
- Click the row of the variant you want to keep, then Keep variant B (the button carries the letter of the selected row).
- Confirm.
The variant you keep is sent to 100% of new leads and the test stops. Its row reads Winner: variant B. The other variants are not deleted: they are paused. If the numbers put another variant ahead, the confirmation window warns you before you confirm.
To start the test again, select the winning row and click Resume the test: every variant returns to the test with an equal share.
Rename or delete a variant
Section titled “Rename or delete a variant”Open Agents, click your agent, open the Prompt tab and the First message step, click the variant button, then Manage the A/B test….
To rename a variant, click the pencil next to its name (Click to rename), type up to 80 characters and press Enter. An empty name brings the letter back, for example Variant B. Publishing replaces that name with the Version name of the publish window: type your name there to keep it. If you keep the date it suggests, the variant shows its letter again.
To delete a variant, click its row, then Delete this variant and confirm with Delete the variant. The link shows only when the test has at least two variants.
The A/B test of the Conversation step
Section titled “The A/B test of the Conversation step”Open Agents, click your agent, open the Prompt tab, then the Conversation step. More options offers the same Test a 2nd version (A/B) entry, with the same window, rows and switches.
- A lead is given a variant when the agent first answers it, and keeps that variant for the whole conversation. If its variant is paused or deleted, the lead gets another one.
- The window adds Qualification after a reply: of the leads who replied under each variant, how many became qualified.
- Each variant can book meetings or not. Open the Settings tab, then Meetings: the block By behavior variant has one switch per variant. See Meetings.
See which variant a lead received
Section titled “See which variant a lead received”- In Conversations, a message written by your agent under a variant carries its letter after the agent’s name, for example
· variant B. Hover it to read the variant’s name. When one agent is selected and its conversation test runs, the filters also offer A/B variant. - On the agent’s Results tab, a line tells you that a test is running, with the link See the test.
- In Analytics, under Compare, the agent’s line reads A/B test running and opens its Prompt tab.
Analytics no longer has an A/B scoreboard: the numbers of a test are read in the Prompt tab, in the window opened by Manage the A/B test….
Why a variant gets no new leads
Section titled “Why a variant gets no new leads”Open Agents, click your agent, open the Prompt tab and the First message step, click the variant button, then Manage the A/B test…, and check the row of the variant.
- It reads Paused: turn its switch on.
- Another row reads Winner: variant A: the test was stopped. Select that row and click Resume the test.
- You changed its text without publishing: a draft is never sent. Open the variant and click Publish.
- It has sent more than the others: for leads coming from a form or the API, the variant that has sent the fewest gets the next leads first, so the one ahead waits.
- No first message leaves at all: the agent is paused, its number is disconnected, or no new lead is coming in. See Agents and Lists & import.