Reading campaign reports
Every campaign has a report that compares its variations against the control on your goals: conversion rate, revenue per visitor, profit per visitor and average order value. The report tells you whether a difference is statistically significant, how close the test is to the sample size it needs, and which variation to deploy. This page walks through each part of the report. The formulas behind the numbers are in Statistics methodology.
Open a report from Campaigns by clicking the campaign name, or from the Campaigns card on the Dashboard.
Report layout
An A/B test report shows, from top to bottom:
Section | What it shows |
|---|---|
Header | Campaign status (change it here), Edit campaign, report filters and the date range |
Outliers detected | Appears when unusually large goal hits may skew results. See Outliers |
Hypothesis | The campaign's description, if you wrote one |
Deployed | Appears after you deploy a winner. Statistics stop being collected from then on |
Historical chart | The comparison metric over time, per variation, in hourly, daily or monthly points in your time zone |
Results | The main table, the test progress and the verdict |
Performance charts | Conversion rate, revenue per visitor, profit per visitor, AOV, visitors and goal hits per variation, with projected impact |
Breakdown by variation/goal | Every goal the campaign tracks, per variation |
The Results table
The table compares each variation against Original (the control) for one goal. By default it shows the campaign's primary goal, compared by its primary metric.
Column | Meaning |
|---|---|
Views | Total campaign views. One visitor can view many times |
Reach | Distinct visitors who saw the control or the variation |
Goals hit | Visitors who reached the goal (default), or every goal hit (see Count visitors with goal hits below) |
Revenue / Profit | Total attributed value of the goal hits |
Comparison metric | Goal CVR, Revenue (or Profit) per visitor, or Average order value |
Confidence | How confident the result is: significance with the frequentist method, probability to beat control with Bayesian |
Confidence interval | A box plot of the interval at the campaign's significance level |
Improvement | Relative change of the comparison metric versus control |
The Confidence cell is highlighted once it reaches the campaign's significance level. Improvement is shown in green when positive and red when negative.
Report controls
Control | What it does | Default |
|---|---|---|
Goal selector | Shows results for another goal the campaign tracks | Primary goal |
Page selector | Limits results to one of the campaign's pages (multi-page campaigns) | All pages |
Compare against | The comparison metric | The campaign's primary metric |
Value metric | Revenue or Profit for value-based metrics | Revenue (Profit when profit tracking is on and the primary metric is profit per visitor) |
Count visitors with goal hits | On: Goal CVR and Goals hit count converting visitors. Off: they count every goal hit | On |
Break down by | Splits the table by Device, Country, Channel or Referrer. See Breakdowns | None |
Individual goal hits
For purchase goals (Shopify Purchase and the custom checkout purchase events), click View events in a row to see that variation's attributed orders: timestamp, value, profit and shipping (if tracked), z-score, the order ID (Remote ID) and the products. With order tagging on, the row menu also offers View orders, which opens Shopify admin filtered to that variation's orders.
Progress and verdict
Below the table, the progress bar shows how much of the required sample the test has collected, based on your significance level, statistical power and minimum detectable effect. It ends in one of three verdicts:
- Needs approximately N more visitors: keep the test running.
- Sufficient data, no significant winner: the test reached its sample size without a winner. The real difference is likely smaller than your MDE.
- Has a winner: a variation beats control on the primary metric at your significance level. You can deploy it from here. See Scheduling, stopping and deploying winners.
The verdict is only reliable for the primary goal with no filters. If you pick another goal or apply filters, progress is hidden with a note explaining why, and you can show it anyway. It's always hidden while a breakdown is active. How the sample size and the winner are calculated is explained in Statistics methodology.
Performance charts and projected impact
Each chart compares one metric across variations and highlights the best performer. The conversion rate, revenue per visitor and profit per visitor charts also project the impact of the best variation. You choose the period under Estimation period: Weekly, Monthly or Yearly.
projected change = (change in metric) × (visitors in the test) × (days in period ÷ days running)For example, if variation B adds $0.40 revenue per visitor, 30,000 visitors have entered the test, and it has run for 15 days, the monthly projection is 0.40 × 30,000 × (30 ÷ 15) = +$24,000. This assumes your traffic and the measured lift stay the same, and that 100% of the test's traffic gets the winner. Treat it as an order of magnitude, not a forecast, especially before the test is significant.
Personalization reports
A personalization report compares Personalized visitors (everyone who saw any variation) with the Control group (visitors who matched a variation's target but were held back).
- Confidence, Confidence interval and Improvement only appear when the campaign has a control group.
- Each variation has its own row with its views, reach, goals and comparison metric.
- Expand Control to see the control group split by the variation each visitor would have seen. Compare each variation against its own slice of the control rather than against the mixed total.
- With a control group, the report shows the same progress bar and verdict as an A/B test, comparing personalized visitors with the control. Without a control group there's no progress bar or verdict. Personalizations can't be deployed.
Export and share
- Export a report card as CSV (choose delimiter, line endings and BOM), PDF (optionally with a summary) or Image (PNG, JPEG or WEBP).
- Print the report for a formatted, filter-annotated version.
- Applied filters are saved in the page URL, so you can share a filtered view with a teammate.
Behavior details and limitations
- Reports refresh every couple of minutes. A conversion that just happened can take a few minutes to appear.
- Date ranges and the attribution window. Exposures are counted inside the date range. Conversions are counted up to the end of the range plus the attribution window, so the last days of a range can still gain conversions. The default range is the campaign's whole lifetime. Presets: Today, Last 24 hours, Last week, Last month, Last quarter and Last year.
- Reports don't add up across campaigns. One order can be credited to several campaigns. See Revenue attribution.
- Don't stop at the first significant result. Significance is recalculated every time you open the report, without a peeking correction. Wait for the progress bar to reach the required sample. See Statistics methodology.
- Uneven reach in a 50/50 test isn't flagged automatically. If reach is visibly unbalanced, check for redirects, caching or targeting that behaves differently per variation before trusting the result.
- Deployed campaigns show the data collected before deployment. The deployed variation isn't tracked.
- Reset stats (Campaigns list → campaign menu → Reset stats) permanently deletes a campaign's data, for all time or a date range. See Report filters, outliers and resetting stats.
Examples
- Price test on profit: primary metric Profit per visitor. Conversion rate drops 4% for the higher price, but AOV rises 11% and profit per visitor rises 7% at 93% confidence with a 90% significance level. The verdict reads Has a winner, and the conversion-rate chart is a secondary signal, not the decision.
- Checking a secondary goal: switch the goal selector from Purchase to Add to cart to see whether a product-page change moved the earlier funnel step. Progress is hidden, because the verdict only applies to the primary goal.
- Segment check before deploying: break down by Device. If mobile and desktop both point the same way, deploy. If one segment is strongly negative, consider a personalization by device instead.