Widget A/B testing
A widget experiment is an A/B test between up to 5 variations of the same popup, bar, callout or embedded widget, optionally with a control group that sees no widget at all. Each visitor is randomly assigned to one variation and keeps it on later visits. The widget report compares variations on conversion rate, goal conversion rate or revenue per visitor, and tells you how confident you can be in the difference.
Available on: all widget types and platforms. Plan: widget A/B testing must be included in your plan.
How it works
- You create an experiment from Experiments in the builder's left toolbar or on the widget page. Its variations start as copies of the widget's current design.
- You edit each variation in the builder: design, steps, triggers and close triggers can all differ. Rules, goals and the schedule are shared by the whole widget.
- You start the experiment, and the widget. Both must be running for visitors to be assigned.
- Visitors are assigned on their first eligible page view: when the widget's page-level rules pass. The assignment is random, weighted by each variation's share, and stored in the visitor's browser. On later pages and visits they get the same variation.
- Each visitor sees their variation when its trigger fires. Visitors in the control group see nothing, but a control view is recorded when the trigger would have shown the widget.
- Results are attributed like any widget's: conversions in the widget, and goals reached afterwards within your attribution window.
- You stop the experiment and pick a winner. Its design is copied to the widget, which then shows it to everyone.
Create an experiment
Option | What it does | Values / default |
|---|---|---|
Name | Shown in the builder and reports | Text |
Primary metric | The metric the experiment is judged on | Conversion rate (widget conversions per view), Goal conversion rate (visitors reaching a goal per visitor reached), Revenue (revenue per visitor). Default Conversion rate |
Variations | Copies of the current design to test | 2 to start; up to 5 |
Control | A holdout group that sees no widget | Off. When on, set its share |
Shares | Percentage of visitors per variation and control | Must add up to 100% |
Even split | Splits traffic equally between the control group and every variation, ignoring the percentages | On |
- With a control group, an experiment needs at least 1 variation. Without one, it needs at least 2.
- Conversion rate isn't available with a control group, because control visitors can't convert in a widget they don't see. Turning on control switches the primary metric to Goal conversion rate.
- Use Even split for 3, 6 or 7 groups: whole percentages can't divide 100 evenly for them.
Variations can be renamed, cloned (up to 5) and deleted from the Experiments panel. Deleting a variation can't be undone.
What differs between variations
On the widget page, triggers and close triggers show a badge with the variation they belong to.
Control groups
The control group measures what happens without the widget, so you can see whether it adds sales or only takes credit for them.
- Control visitors see nothing. A control view is recorded when any trigger used by any of the variations fires and the rules pass, so control is exposed at the same moment variations would be.
- Control views count toward the visitor's Frequency rule, as if they had seen the widget.
- Control visitors are credited with goals they reach within the attribution window, like variation visitors.
- For embedded widgets, control visitors see the spot's original content.
With a control group, turn on Attribute on view in the widget's Goals. Otherwise control visitors are credited with every goal they reach, while variation visitors are credited only after converting in the widget, which makes variations look worse than they are. The report shows a warning when this applies.
Reading the results
Open the widget's report. The Experiments card compares the variations of the selected experiment:
Column | Meaning |
|---|---|
Views | Times the variation was shown (blank for control) |
Reach | Unique visitors who were shown the variation, or assigned to control |
Clicks, Submits | Clicks on buttons, linked images and products, and form submits |
Goals hit | Goal hits credited to the variation |
Revenue | Value of the goal hits credited to the variation |
Metric | The comparison metric you choose |
Improvement | Relative difference against the control, or against the lowest-performing variation if there's no control |
Comparison metrics:
Metric | Formula |
|---|---|
Widget CVR | Conversions ÷ views |
Goal CVR | Visitors with goal hits ÷ reach |
Completion rate | Completions ÷ views |
Revenue | Total attributed value |
Revenue per visitor | Attributed value ÷ reach |
Below the table, the report states how confident you can be that the best variation beats the control (or the lowest-performing variation), for example "We can be 96.40% confident that the Variation 2 variation performed better than the control." Until there's enough data, it says the results are still inconclusive. See Statistics methodology.
Stop and pick a winner
- Pause stops assigning and showing variations. While an experiment is paused, the widget shows its original design to everyone.
- Stop ends the experiment and opens Select winner. The Winning variation you choose is saved to the widget, replacing its original design. You can also keep the original.
- Only one experiment runs on a widget at a time. Experiments can be deleted only while they're drafts.
Behavior details and limitations
- Assignment is per browser. A visitor on a new device or browser, in a private window, or after clearing site data can get a different variation.
- Changing shares affects new visitors only. Visitors already assigned keep their variation. Visitors whose variation was deleted are reassigned.
- Rules are shared, so you can't test audiences. To compare showing a widget to two audiences, use two widgets, or test the widget against a control group within each audience.
- Different triggers per variation are allowed, which lets you test "exit intent vs 20 seconds". The control group is exposed when any of those triggers fires.
- Stats count non-control events. Views, clicks and submits on the widget's totals exclude the control group; goal attribution includes it.
- Analytics integrations receive the experiment. Widget events sent to Google Analytics 4 include the experiment and variation IDs. See Google Analytics 4 and GTM.
Examples
- Discount size: variation 1 offers 10% off, variation 2 offers free shipping. Primary metric Revenue, with Track widget conversion on the form.
- Does the popup help at all? One variation plus a 50% control group, Attribute on view on, primary metric Goal conversion rate on Purchase.
- Timing: two identical variations, one triggered on exit intent and one after 15 seconds.