Statistical testing: sample size, lift and full factorial designs

A green dashboard is not a result

Picture Workbench, an invented Rails SaaS for joiners, plumbers and electricians, run from Leeds at workbench.example. I use it as the example business in this chapter. Its founders look at a slide: "New pricing page headline: plus 32% trial signups." The test ran nine days: 41 trials from 1,050 visitors on the old page, 54 from 1,050 on the new one. They rewrite every headline on the site to match. Two months later the trial rate is exactly where it was, and nobody can say why.

Run the numbers they skipped. A two proportion test on 41 out of 1,050 against 54 out of 1,050 gives $z = 1.37$ and $p = 0.17$ , and the 95% interval on the difference runs from minus 0.54 to plus 3.02 percentage points. The "plus 32%" was compatible with the new headline doing nothing at all. Nothing dishonest happened. The team had a tool that splits traffic and a dashboard that turns green, and they stopped when it turned green. What they did not have was a number written down before the start: how big a lift they needed to see, how many visitors that takes, and what they would do if the answer was "nothing". That is this chapter. The maths is not hard. The discipline is.

Where this sits in the series

This is chapter 20 of 21 and the first of part five, testing and big data. Part four closed with the customer loyalty journey, where a loyalty programme turned out to be a series of hypotheses about what a segment will do next. Testing is how you find out whether those hypotheses were right, and it turns every earlier chapter, from elasticity to media mix, from a model into a decision. The series ends with big data for marketers. For the traffic and budget side of the same question (how many tests your visitors can carry at all), read how much data you actually need first. Every sample size in this article uses the same formula as that article and the planner on my data science page, so the figures match to the visitor.

Lift first, then sample size

Start with what you actually care about. Not "is B different from A" but "is B better than A by enough to matter". The quantity is the lift (relative change, see percentages):

\text{lift} = \frac{p_B - p_A}{p_A}

where $p_A$ is the conversion rate of the control (the page you have today) and $p_B$ the conversion rate of the variant. Lift is relative: a move from 4.0% to 4.6% is a 15% lift, though the absolute gap is only 0.6 percentage points. Keep both in your head, because the sample size depends on the absolute gap while the business case is argued in relative terms.

The lift you need to detect is a business number, not a statistical one. For Workbench: 2,000 visitors a month reach the pricing page and 4% start a trial, so 80 trials. A quarter of trials become paying customers at £79 a month, and a customer stays about 22 months, so each one is worth about £1,738 in revenue. A 15% lift in trials is 12 extra trials a month, 3 extra customers, about £5,200 of lifetime revenue added for every month the better page runs. That is the smallest effect worth a redesign and a test. Below it, whether the result is "significant" does not matter. This is the minimum detectable effect (MDE), and choosing it is the most important decision in the design of a test. Why statistical significance and error margins matter covers the reading of a result. Here I go one step earlier, to the moment before the test exists.

The sample size equation

You can make two mistakes when you read a test. You can declare a winner when there is none (a false positive, probability $\alpha$ ), and you can miss a real winner (a false negative, probability $\beta$ ). The conventions are $\alpha = 0.05$ two sided and $\beta = 0.20$ , a power of 80%. They are conventions, not laws.

For two conversion rates the visitors needed in each arm are (derivation in sample size):

n = \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2\,\bar p\,(1-\bar p)}{(p_B - p_A)^2}, \qquad \bar p = \frac{p_A + p_B}{2}

where $n$ is the number of visitors per arm, $p_A$ the baseline rate, $p_B = p_A(1 + \text{lift})$ the rate you hope to detect, $\bar p$ their average, $z_{1-\alpha/2} = 1.96$ the standard normal point for a two sided 5% test and $z_{1-\beta} = 0.84$ the point for 80% power. The denominator is the squared absolute difference, and that square is why small lifts on small baselines are so expensive: halve the gap you want to detect and you need four times the visitors. Some calculators use a slightly different variance term under the alternative. At these rates the two versions differ by a handful of visitors, and I use this one because the planner and the sibling article do.

Plug in Workbench. Baseline 4%, target 4.6%, gap 0.006, average 4.3%. The constant $2(1.96 + 0.84)^2$ is 15.698 and $\bar p(1 - \bar p) = 0.043 \times 0.957 = 0.041151$ , so

n = \frac{15.698 \times 0.041151}{0.006^2} = 17{,}944

visitors per arm ( $n$ ), 35,888 in total, 18 months at 2,000 pricing page visitors a month. The nine day test with 1,050 per arm could only reliably detect a lift of about 69%. It was never a test of a headline. It was a coin flip with a dashboard.

A sample size table you can use

Here is the equation for the baselines and lifts that come up most often: visitors per variant, two sided 5%, 80% power, rounded up to the next whole visitor.

Baseline rate	5% lift	10% lift	15% lift	25% lift	50% lift
1%	637,012	163,096	74,195	27,939	7,751
2%	315,208	80,683	36,694	13,811	3,827
4%	154,306	39,476	17,944	6,747	1,865
8%	73,855	18,873	8,569	3,215	883
15%	36,311	9,258	4,194	1,566	426

Read it row by row. A 1% baseline is a purchase from cold traffic on a landing page; a 5% lift there needs more than 600,000 visitors per arm, which for a small business means never. At 4%, a typical trial or lead form rate, the 15% cell is the Workbench answer. At 15%, an add to basket rate or the click rate of a warm newsletter list, a 25% lift is detectable with 1,566 per arm, a fortnight for most shops. The lesson: test where the baseline is high and the lever is big, further up the funnel and on bolder changes. Micro changes to low rate steps are not testable for small businesses, and no tool changes that.

A/B tests versus full factorial designs

An A/B test changes one thing. It is clean and slow, and its deeper problem is that a pricing page is not one thing. Headline, price anchor and proof all pull on the same visitor at the same time. Testing them one after another takes three times as long and says nothing about how they combine. Why testing one thing at a time is costing you money makes that argument in full, so here I only show the alternative.

A full factorial design tests every combination of every factor.¹ Three factors at two levels each is a two by two by two, eight cells, and every visitor is randomised into one of them. What makes it affordable is that each main effect is estimated from the whole sample: to measure the headline you compare the four cells with headline one against the four with headline two, 3,000 against 3,000 in a 6,000 visitor test, exactly as in an A/B test of the headline alone. For main effects, three factors cost no more visitors than one, and the interactions come with them.

The design matrix behind it

Write each factor as a column of plus one and minus one, one row per cell. Three factors give a matrix with eight rows and three columns, and every pair of columns is orthogonal: their products sum to zero. That orthogonality is what lets one sample answer three questions without the answers leaking into each other (the algebra is in matrices). The main effect of factor $j$ is

\text{ME}_j = \frac{1}{2^{k-1}} \sum_{i=1}^{2^k} x_{ij}\,\bar y_i

where $k$ is the number of factors (here 3, so $2^k = 8$ cells), $x_{ij}$ is plus or minus one depending on whether cell $i$ has factor $j$ switched on, and $\bar y_i$ is the conversion rate in cell $i$ . It is simply the average of the four "on" cells minus the average of the four "off" cells. An interaction is the difference of differences: does the anchor help more when proof is present than when it is absent?

\text{INT}_{A \times P} = \left(\bar y_{A,P} - \bar y_{N,P}\right) - \left(\bar y_{A,N} - \bar y_{N,N}\right)

where the first subscript is anchor ( $A$ ) or no anchor ( $N$ ), the second is proof ( $P$ ) or no proof ( $N$ ), and each $\bar y$ is pooled over both headlines (1,500 visitors). In matrix terms the interaction column is the product of the anchor and proof columns, and it too is orthogonal to everything else (see matrices).

The Workbench factorial: eight cells, 6,000 visitors

Workbench runs the pricing page as a two by two by two over a quarter. Headline: the current feature headline ("Scheduling, quotes and invoices in one place") or an outcome headline ("Get your evenings back"). Price anchor: the monthly price alone, or the price next to the crossed out cost of the three tools it replaces. Proof: nothing, or a strip of customer logos and one named quote from a joiner in Bristol. The numbers are invented, 750 visitors per cell.

Cell	Headline	Anchor	Proof	Visitors	Trials	Rate
FNN	Feature	None	None	750	30	4.00%
FNP	Feature	None	Logos	750	36	4.80%
FAN	Feature	Anchor	None	750	25	3.33%
FAP	Feature	Anchor	Logos	750	50	6.67%
ONN	Outcome	None	None	750	34	4.53%
ONP	Outcome	None	Logos	750	40	5.33%
OAN	Outcome	Anchor	None	750	29	3.87%
OAP	Outcome	Anchor	Logos	750	54	7.20%

FNN is the page as it was, 4.0%, the baseline used for the sample size. FNP adds proof and moves to 4.8%. FAN adds the anchor without proof and drops to 3.33%: a crossed out competitor price with nothing behind it reads as a sales trick. FAP, anchor plus proof, jumps to 6.67%, far more than the two single changes add up to. The outcome headline rows repeat the shape about half a point higher, ending with the best cell, OAP, at 7.20%.

Now the main effects, each from 3,000 against 3,000 visitors:

Factor	Off	On	Difference	z	p value	95% interval on the difference
Headline (feature, outcome)	4.70%	5.23%	+0.53 pts	0.95	0.34	minus 0.57 to +1.63 pts
Anchor (none, anchor)	4.67%	5.27%	+0.60 pts	1.07	0.28	minus 0.50 to +1.70 pts
Proof (none, logos)	3.93%	6.00%	+2.07 pts	3.68	0.0002	+0.97 to +3.16 pts

Proof is the only clear main effect: 2.07 points, a relative lift of 53%, with an interval well clear of zero. The headline, the thing the founders were so excited about, shows half a point with an interval from minus 0.57 to plus 1.63. That is not "the headline does nothing". It is "6,000 visitors cannot tell you what the headline does", a different and more honest reading. With 3,000 per side at a 4% baseline the sample size formula says the smallest reliably detectable lift is about 39%, roughly 1.6 points. Proof cleared that bar, the headline and the anchor did not. The anchor's main effect is a shrug at plus 0.6 points, and that is where the factorial earns its keep.

Reading the interaction

Look at the anchor in each condition. Without proof, adding it moves the rate from 4.27% to 3.60%, minus 0.67 points. With proof, it moves the rate from 5.07% to 6.93%, plus 1.87 points. The difference of those differences is 2.53 points. Its standard error is the square root of the sum of $p(1-p)/1{,}500$ over the four pooled cells, 1.12 points, so $z = 2.26$ and $p = 0.024$ .

The matrix view says the same thing more compactly. The interaction column applied to the cell rates gives 1.27 points, exactly half the difference of differences: the anchor's average effect is plus 0.60, it sits 1.27 above that average when proof is present (plus 1.87) and 1.27 below it when proof is absent (minus 0.67). The anchor is not a weak lever. It points in opposite directions depending on whether the page has earned the right to make a price comparison, and averaged over both conditions the halves cancel into a meaningless plus 0.6.

Had Workbench tested the anchor alone on the existing page, they would have measured a drop, concluded that anchoring does not work for tradespeople and never tried it again. Had they tested proof alone they would have shipped it and left the anchor's contribution on the table. The interaction is the finding, and only a design that varies both factors at once can see it.

The caveat matters. Each side of the interaction is a 1,500 against 1,500 comparison, so its standard error is twice that of a main effect, and detecting an interaction as large as a main effect takes four times the visitors. That is the same arithmetic behind the rule in how much data you actually need that a 2 by 2 test needs four times the visitors per arm when you want to read the cells. At $p = 0.024$ this interaction is real enough to act on and not certain enough to build a pricing philosophy on. I would ship OAP and rerun anchor against no anchor on the new page with proof, as a confirmation.

Reading results: the interval, not the verdict

The two proportion z test is the standard way to compare two rates (the logic is in hypothesis tests):

z = \frac{\hat p_B - \hat p_A}{\sqrt{\hat p\,(1-\hat p)\left(\dfrac{1}{n_A} + \dfrac{1}{n_B}\right)}}, \qquad \hat p = \frac{x_A + x_B}{n_A + n_B}

where $\hat p_A$ and $\hat p_B$ are the observed rates, $x_A$ and $x_B$ the conversion counts, $n_A$ and $n_B$ the visitors per arm and $\hat p$ the pooled rate under the assumption that nothing differs. A $z$ beyond 1.96 either way is "significant at 5%", the least interesting thing the test can tell you.

The interesting thing is the confidence interval on the difference (confidence intervals):

(\hat p_B - \hat p_A) \;\pm\; z_{1-\alpha/2}\,\sqrt{\frac{\hat p_A(1-\hat p_A)}{n_A} + \frac{\hat p_B(1-\hat p_B)}{n_B}}

where the symbols are as above and $z_{1-\alpha/2} = 1.96$ for a 95% interval. For proof that is 2.07 points plus or minus 1.10, so 0.97 to 3.16 points, or in relative terms a lift somewhere between about 25% and 80%. That width is the true state of your knowledge. Report "plus 53%" and the finance director will plan on it. Report "between 25% and 80%" and she will plan on the low end, which is correct. Every decision memo I write carries the interval, and where it includes zero the first line says so.

Running a test properly

The protocol is short.

Three details matter more than the diagram suggests. Randomise by visitor, not by day or week, because Tuesdays convert differently from Saturdays. Check the split on day one: if 750 per cell was the plan and one cell has 640, the assignment is broken and every number downstream is suspect. Kohavi, Tang and Xu call this a sample ratio mismatch and treat it as the first check on any experiment.² And pick one primary metric before you start: trials started, not trials plus pricing clicks plus scroll depth. Secondary metrics explain why. They never declare a winner.

Peeking at the dashboard

The most common way honest teams fool themselves is by watching the dashboard.³ The statistic wanders as data comes in, and if you look every day for a month and stop the first day it crosses 1.96, you are not running one test at 5%, you are running thirty and taking the best. You can confirm it by simulating A/A tests, where both arms share the same true rate:

import numpy as np
rng = np.random.default_rng(1)
p, n_day, days, hits = 0.04, 200, 30, 0
for _ in range(2000):
    a = rng.binomial(1, p, n_day * days).cumsum()
    b = rng.binomial(1, p, n_day * days).cumsum()
    for d in range(1, days + 1):
        n = d * n_day
        pa, pb = a[n-1] / n, b[n-1] / n
        pp = (pa + pb) / 2
        se = np.sqrt(pp * (1 - pp) * 2 / n)
        if se > 0 and abs(pb - pa) / se > 1.96:
            hits += 1
            break
print(hits / 2000)   # the share of A/A tests that "found a winner"

I ran the same setup 40,000 times: reading once at the end produced a false winner in 4.9% of tests, ten looks (every third day) in 19%, and a daily look for 30 days in 28%. These match the simulation in how much data you actually need, which also gives the thresholds for a planned sequential design. The cure is either to fix the sample size in advance and read once at the end, which is what I do for most small business tests, or to use a proper sequential method that spends the error budget across looks, such as an O'Brien and Fleming type plan or the always valid p values of Johari, Pekelis and Walsh.⁴ That is fine if your tool implements one and you know which. A tool that turns green on day nine is not doing that.

When a test is not worth running

Declining a test is often the right call. A test is not worth running when the required sample exceeds what the page will see in about a quarter, because by then the season, the ad mix and the product have all changed under it. Not when the change is obviously right and reversible: fixing a broken form field needs no control arm. Not when the value of the information is smaller than the cost of delaying the change, as with small copy edits on low traffic pages. And not when the business has already decided what it will do regardless of the result. In that case the honest move is to say so and spend the effort on something that can still change a decision.

The alternative to a test is not guessing. It is the earlier chapters of this series: a regression on historical data, a survey, a well read funnel. Tests are the gold standard for causation and also the most expensive instrument in the box. The guide to digital marketing testing methods walks through the less expensive instruments and when each one is enough.

Running it yourself

You need three things: a randomisation mechanism that assigns a visitor once and remembers it (a first party cookie or a hashed user id, set within your consent rules), an event that records the primary conversion against that assignment, and a way to count. Any tool works, and a spreadsheet works for the reading. The sample size in a few lines, the same formula as above and as the planner:

from math import ceil
from statistics import NormalDist
N = NormalDist()

def sample_size(p_a, lift, alpha=0.05, power=0.8):
    p_b = p_a * (1 + lift); p_bar = (p_a + p_b) / 2
    z = N.inv_cdf(1 - alpha / 2) + N.inv_cdf(power)
    return ceil(2 * z ** 2 * p_bar * (1 - p_bar) / (p_b - p_a) ** 2)

print(sample_size(0.04, 0.15))   # 17944

Before you trust a result, run four checks. The split: are the cell sizes within what randomisation should give? The A/A check: do two arms of the control differ by less than the noise predicts? The segments: does the effect hold on mobile and desktop, new and returning, or is it driven by one burst of bot traffic on one day? The calendar: did a newsletter, a price change or a bank holiday in Scotland but not in England hit the arms unequally? Only after those do I look at the interval.

How long does it take? Designing a factorial test is an afternoon. Running it is the sample size divided by daily traffic, and the honest number is often longer than anyone likes. Reading it is an hour and a memo.

Pitfalls

Testing to vanity precision. "How many visitors do I need for the result to be accurate?" usually gets answered with a margin of error on a single rate. Wrong question. Precision on a rate is not power to detect a difference, and the difference you need is a business number you have to choose. If you have not chosen it, you have not designed a test.

Stopping on green. Covered above, and still the most common failure. If your tool lets you stop early without a sequential correction, decide the end date and hide the dashboard.

Too many cells. Four factors at three levels is 81 cells, and 6,000 visitors gives about 74 per cell. Main effects still pool, but the interactions become noise. Two or three factors at two levels suits small business traffic. Fractional factorial designs give up some higher order interactions to save cells, deliberately confounding them with other effects,⁵ a fair trade when you know those interactions are unlikely.

The novelty effect. A new headline can do better in week one simply because returning visitors notice the change, and the effect fades as they get used to it.² Run long enough to cover full weekly cycles and look at new visitors separately. If the effect shrinks week on week, you have measured curiosity, not preference.

The wrong metric. Trial starts went up 53%. Did paying customers? Proof may attract trials from people who were never going to buy. Follow the primary metric to the money whenever the lag allows, and otherwise check that the downstream rates did not move the other way.

Mistaking a losing test for a wasted test. FAN, anchor without proof, lost. That is one of the most useful things Workbench could learn, because it explains why an earlier anchoring experiment failed and warns every future designer. A well designed test that returns "worse" has paid for itself.

How I do this for clients

The deliverable is a test programme, not a single test. It starts with the free workshop, where we list every change anyone wants to make to the key page, put a lift and a sample size against each one, and cross out the ones the traffic cannot support. Most such lists shrink a lot in that hour, and that is the point.

Then the first two weeks of work are on me, before any commitment. I need access to the analytics and the page, the historical baseline for the primary metric by day and by segment, and whoever owns the design of the variants. In those two weeks I write the test plan as a one page document (hypothesis with a number, primary metric, factors and levels, sample size, end date, stopping rule and what we will do with each possible result), set up the randomisation and the event, run an A/A check and start the first factorial.

At the end of a test you get a decision memo with the lift, its interval and a risk statement in the first paragraph, the full cell table behind it, and a recommendation you can act on that week. Over a programme you get a running register of what was tested, what it did and what it taught. Your raw data stays yours, and the terms for code and models are in the analytics and data science terms.

This sits inside my data science work and, where the tests are on ad landing pages, inside performance marketing. The pricing page shows how the work is priced. What I will not do is report a winner without an interval.

Questions to ask before your next test

What lift did we decide, before the test, was the smallest worth detecting, and where is that written down? How many visitors per arm does that lift need at our baseline, and when will we reach it? Which one metric decides the test, and who agreed to it in advance? Is the assignment random by visitor, and did we check the split on day one? Are we reading the result once at the planned end, or has someone been watching the dashboard? What is the confidence interval on the lift, and does the decision still hold at its lower end? Could the factors we are testing interact, and if so why are we testing them one at a time? And what will we do if the result is "no difference"?

The series is inspired by Mike Grigsby's Marketing Analytics.⁶ The explanations, examples and numbers here are my own.

NIST/SEMATECH e Handbook of Statistical Methods, section 5.3.3.3.1, Two level full factorial designs, https://www.itl.nist.gov/div898/handbook/pri/section3/pri3331.htm ↩
Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020 (sample ratio mismatch, novelty and primacy effects), https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59 ↩ ↩²
Evan Miller, How Not To Run an A/B Test, https://www.evanmiller.org/how-not-to-run-an-ab-test.html ↩
Ramesh Johari, Leo Pekelis and David J. Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing, arXiv:1512.04922, https://arxiv.org/abs/1512.04922 ↩
NIST/SEMATECH e Handbook of Statistical Methods, section 5.3.3.4.3, Confounding (also called aliasing), https://www.itl.nist.gov/div898/handbook/pri/section3/pri3343.htm ↩
Mike Grigsby, Marketing Analytics: A Practical Guide to Improving Consumer Insights Using Data Techniques, Kogan Page, 2022, ISBN 9781398608191, https://www.netgalley.co.uk/catalog/book/267990 ↩