A/B tests (experiments)
An A/B test (called an Experiment in Personizely) randomly splits your visitors between the original version of your site, the control, and one or more variations. It measures which version earns more conversions, revenue or profit per visitor. You can A/B test page content, URLs (split URL tests), Shopify themes, templates, product prices and shipping rates, or your own code with full-stack experiments.
Available on: all platforms. Theme, template, price and shipping tests are Shopify only. See Campaigns.
How it works
- A visitor reaches a page where the test applies. The page must match the campaign's pages (for content and URL split tests), and the visitor must match the target, if you set one.
- Traffic allocation is checked. If allocation is below 100%, only that share of eligible visitors enters the test. The others see the original and aren't counted.
- The visitor is assigned to control or a variation, according to the split. A view is recorded for that arm.
- The assignment sticks. On later pages and later visits, the visitor keeps seeing the same version. See How visitors are assigned to variations.
- Conversions are credited to the arm the visitor saw, within the attribution window. Control visitors are credited too; they are the baseline.
- The report compares each variation to control on your primary metric and declares a winner once the result is significant. See Statistics methodology.
Set up an A/B test
- Go to Campaigns → New campaign, choose Experiment, then the type.
- Define the first variation. For content tests you edit it in the visual editor; for other types you pick the URL, theme, template, prices or shipping rates.
- In the campaign editor, add more variations with Create variation if you need them.
- Choose the target: All visitors or Specific target.
- Add goals, mark the primary goal and choose the Primary metric.
- Check Advanced settings (significance level, methodology, power, MDE).
- Preview and QA each variation, then Start or Schedule the campaign.
Option | What it does | Values / default |
|---|---|---|
Control | The original version of your site. It's always there in an experiment | Share default 50% with one variation |
Variations | The versions you're testing against control | One or more. Drag to reorder |
Even split | Splits traffic equally between control and every variation, whatever their number | On for new campaigns |
Shares | With Even split off, the share of traffic for control and each variation. Shares always total 100%; changing one adjusts the others | 0–100% each |
Target | All visitors, or a Specific target that limits who can enter the test | All visitors |
Mode (with a specific target) | Live: the target is checked on every page. Permanent: once a visitor has been assigned, the target isn't checked again for them | Live. Price tests always use Permanent |
Traffic allocation | Share of eligible visitors who enter the test. The rest see the original and aren't counted | 0–100%, default 100% |
Goals | What counts as a conversion. One goal is the primary goal | At least one required |
Primary metric | The metric used to compare variations and call the winner | Conversion rate, Revenue per visitor, Profit per visitor, Average order value |
Trigger (content tests) | Immediate runs as the page loads. Manual runs only when your code calls ply.runCampaign(id) | Immediate |
Significance level | Confidence required to declare a winner | 80%, 90% (default), 95%, 100% |
Statistical methodology | The statistical test | Frequentist (default), Bayesian |
Statistical power | Chance of detecting a real effect of at least the MDE | 60–99%, default 80% |
Minimum Detectable Effect (MDE) | Smallest relative lift you want to detect; sizes the test | 1–20%, default 20% |
Reload the page for the control group (URL split and template tests) | Gives control visitors the same extra page load that variation visitors get from the redirect, so both arms are counted the same way | Off |
Behavior details and limitations
- Even split handles any number of arms. Percentages can't divide exactly by 3, 6 or 7, so use Even split instead of typing 33/33/34 by hand.
- Changing the split mid-test only affects new visitors. Visitors already in the test keep their arm. The new split applies to visitors entering from then on, so the arms' traffic mix shifts over time.
- Lowering traffic allocation mid-test removes visitors. Allocation is re-checked on every page, so some visitors who were already in the test stop seeing their variation. Raising it lets new visitors in. Set allocation before you start and leave it.
- Live vs Permanent targeting. With Live, a visitor who stops matching the target (for example, their cart value drops below a threshold) sees the original on those pages but keeps their assignment, and sees their variation again when they match. With Permanent, they keep seeing their variation once assigned.
- Adding variations is limited once traffic arrives. Content, URL split, price, shipping and full-stack tests can't get new variations after 1,000 views, because the new arm would start far behind the others. Clone the campaign to test a new idea.
- Deleting a variation reassigns its visitors. Visitors who were in it are assigned again among the remaining arms, including control. See How visitors are assigned to variations.
- Redirect tests and the control group. In URL split and template tests, a variation is counted after its redirect loads, so variation visitors who leave during the redirect aren't counted, while control visitors are. Reload the page for the control group removes that bias at the cost of one extra page load for control visitors. Theme tests redirect only once per preview session and don't need it.
- Manual trigger. A content test with the Manual trigger runs only when your code calls ply.runCampaign(id). After a visitor has been shown a variation, it runs automatically for them on later pages. See JavaScript API (window.ply).
- Overlapping tests. Several tests can run on the same page. If two content tests change the same element, the first one to apply wins. Put tests in a campaign group when each visitor should only see one.
- Winners can be deployed. Content, URL split, theme, template and price tests can roll the winning variation out to all visitors. Shipping and full-stack tests can't. See Scheduling, stopping and deploying winners.
Examples
- Product page layout: a content test that moves reviews above the fold, on the product page template, all visitors, primary metric revenue per visitor.
- Three-way headline test: control plus two new headlines with Even split on, so each arm gets one third of traffic.
- Free-shipping upsell test: a Specific target for visitors with a cart value over $40, with Mode set to Permanent, so visitors who remove an item keep seeing the variation they were assigned.
- Cautious rollout: a risky checkout-adjacent change with Traffic allocation at 20% and a 50/50 split, so only 10% of visitors see the variation.
Troubleshooting
- Reach is very uneven in a 50/50 test → look for something that treats arms differently: a redirect that some visitors abandon (turn on Reload the page for the control group), a target that matches differently per variation, or a page cache that serves the same content to everyone.
- A variation never shows for me → you may already be assigned to control. Open the site in a private window, or use a preview link to see a specific variation.
- Views stopped after editing the campaign → check the schedule, traffic allocation and target. The campaign may be paused or ended, or the new target may exclude most visitors.