Statistics methodology
Personizely compares each variation of an A/B test against the control to decide whether a difference is real or just noise. It does this for conversion rate, revenue per visitor, profit per visitor and AOV. By default it uses a frequentist test at 90% significance, and you can switch any campaign to Bayesian. This page explains the formulas, the settings, and how a winner is declared.
What's being compared
Each row in a campaign report is the control or a variation, measured on these numbers:
Metric | Definition |
|---|---|
Reach | Distinct visitors who saw the control or variation |
Views | Total views (one visitor can view many times) |
Conversions | Goal events attributed to the row |
Converting visitors | Distinct visitors with at least one attributed conversion |
Conversion rate | Converting visitors ÷ reach |
Revenue per visitor | Attributed revenue ÷ reach |
Profit per visitor | (Attributed profit [+ shipping, if enabled]) ÷ reach |
AOV | Attributed revenue ÷ attributed orders |
Conversion rate counts visitors, not events. A visitor who buys three times counts once toward conversion rate, but all three orders count toward revenue. Per-visitor metrics divide by every visitor who was exposed, including those who never bought. They therefore capture changes both in how many people buy and in how much they spend.
Conversions are credited using your Revenue attribution (7 days by default). When you report on a date range, conversions are counted up to the end of the range plus the window, so visitors exposed near the end of the range still get credit.
Statistical settings
You set these per campaign under Advanced settings:
Setting | What it does | Values / default |
|---|---|---|
Primary metric | The metric used to call a winner and size the test | Goal conversion rate, Revenue per visitor, Profit per visitor, AOV |
Methodology | The statistical test | Frequentist (default), Bayesian |
Significance level | How confident the result must be before a variation is called a winner | 80%, 90% (default), 95%, 100% |
Statistical power | Chance of detecting a real effect of at least the MDE | 60–99%, default 80% |
Minimum detectable effect (MDE) | The smallest relative lift you care about detecting; drives the sample size estimate | 1–20%, default 20% |
Frequentist method (default)
- Conversion rate: a two-sided z-test for the difference between two proportions, using the unpooled standard error.
- Revenue, profit and AOV: a z-test on the difference of means. Per-visitor variance is computed exactly from each visitor's values (sum of squares). For older data that lacks them, it is estimated from conversions.
- Significance is shown as (1 − p-value) × 100. A variation with 94% significance has a p-value of 0.06.
- Confidence intervals are calculated at the campaign's significance level. At the default 90%, you see a 90% interval, not 95%. The results table shows the interval of the difference from control; the box plot shows each variation's own interval.
Bayesian method
- Conversion rate: each row gets a Beta(1, 1) prior (uniform), updated with its conversions and reach. The probability that the variation beats control, P(B > A), is computed in closed form.
- Revenue, profit and AOV: a normal approximation of each row's mean, sampled 10,000 times (Monte Carlo) to estimate P(B > A).
- With Bayesian selected, the significance column shows the probability to beat control. A variation wins when that probability reaches the significance level you set.
Improvement and baseline
- Improvement = (variation − control) ÷ control, for the selected metric.
- The baseline is the control. In a personalization without a control group, the worst-performing variation is used as the baseline.
Sample size and test progress
Personizely estimates how many visitors the test needs from four inputs: the control's current rate (or mean and variance for revenue), your significance level, your power and your MDE. It adjusts for unequal splits, for example a 20% control. Early on, with very little data, the estimate is stabilized so it doesn't swing wildly.
The progress bar is total reach ÷ required sample size. It reads as one of three states:
- Needs approximately N more visitors: keep the test running.
- Sufficient data, no significant winner: you've reached the sample size, but no variation passed the significance level. The difference is likely smaller than your MDE.
- Has a winner: a variation passed the significance level with a positive improvement. You can Scheduling, stopping and deploying winners.
Progress is hidden when you apply a breakdown. When filters are active or you view a goal other than the primary goal, it's hidden with a note explaining why, and you can show it anyway; treat it as approximate.
How a winner is declared
A variation is a winner when both of these are true:
- Its improvement on the primary metric is positive.
- Its significance (or probability to beat control) is at or above the campaign's significance level.
If more than one variation qualifies, the one with the highest improvement is shown first. With the significance level set to 100%, no winner is ever declared. Use it when you only want to watch results.
Behavior details and limitations
- No peeking correction. Significance is recalculated every time you open the report, with no sequential-testing adjustment. If you check daily and stop at the first significant result, your real false-positive rate is higher than the level you set. Decide your sample size up front (the progress bar does this for you) and wait for it.
- No multiple-comparison correction. With several variations, each one is compared to control on its own. More variations means more chances of a false winner. Consider a stricter significance level for tests with three or more variations.
- Breakdowns and filters re-run the statistics per segment. Smaller segments have wider intervals. A "significant" result in one segment out of ten is often noise.
- Outliers affect revenue metrics. One very large order can swing revenue per visitor. You can review and remove statistical outliers per campaign (orders with a z-score above a threshold). Removal is permanent. See Report filters, outliers and resetting stats.
- Reports refresh every couple of minutes. Results are cached briefly, so a conversion that just happened may take a few minutes to appear.
- Sample ratio isn't checked automatically. If a 50/50 test shows a visibly uneven split in reach, investigate before trusting the result: look for redirects, caching, or a target that behaves differently per variation.
Examples
- Conversion-rate test: control 2.0% (400 of 20,000), variation 2.3% (460 of 20,000). Improvement +15%. With frequentist at 90%, the z-test gives roughly 96% significance, so the variation wins.
- Revenue test with a lower rate: a price increase lowers conversion rate by 5% but raises AOV by 12%. Revenue per visitor captures both effects, so it's the metric to call the test on, not conversion rate.