Incrementality testing: how to know which sales your ads bring

What is an incrementality test?

An incrementality test splits your audience into two statistically similar groups. One can see your ads and the other cannot. At the end of the period you compare how much each group bought, and the difference is what the advertising added.

What makes this different from everything else is that it measures against a world without ads. It is the only way to answer the question that matters: what would have happened had I not spent.

Why attribution cannot answer this

Attribution shares credit for a sale among the ads involved along the way. It does its job well, but its job is a different one.

Picture somebody who had already decided to buy from you, searches your brand on Google, clicks your ad and buys. Attribution books that sale to the brand campaign. Incrementality would ask: without that ad, would they have bought anyway through the organic result? Almost always, yes.

That is where the enormous gaps come from. In the Advantage+ case PPC Land reported, an incrementality test showed the system produced only 17% of the conversions Meta’s attribution claimed. We analysed it in the article on Meta’s GEM model.

How does it work underneath?

The mechanism is a clinical trial, applied to advertising.

1. The target audience for the test is defined.
2. The platform splits it at random into test and control groups, balancing age, location, past behaviour and other variables.
3. The test group sees the ads; the control group is held back from exposure.
4. At the close of the period, conversions in both groups are compared.

Randomness in the split is what makes the result valid. If the groups were chosen on any other basis, the difference might come from that basis rather than from the ad.

Three ways to set it up

The user-level test is what the platforms offer. They split people into two groups and withhold exposure from one. It is the cleanest and demands the most volume.

In the geographic test, the split runs by region rather than by person. The campaign is switched off in some provinces and kept in comparable ones. It works when the platform offers no user-level test, and carries an important advantage. It measures total sales, not attributed conversions.

And the time-based test switches on and off over alternating periods. It is the cheapest and the most exposed to seasonality, so it needs long periods and comparison against the same period last year.

For a mid-sized European business, the geographic test is usually the realistic one. It needs no thousands of conversions and runs on the campaigns that already exist.

What volume is required

This is where most accounts fall out, and it is worth knowing before getting excited.

For Meta’s conversion tests, third-party documentation mentions two requirements. An active account with at least $5,000 of spend in the past year, and 500 optimised conversions with the pixel properly configured. Those come from tool vendors rather than an official Meta table.

The real limit is statistical and harsher. Several consultancies publish similar sample-size calculations: detecting a 20% relative lift needs around 475 purchases per arm; a 10% lift, roughly 1,730. With few sales a month, the test cannot separate effect from noise.

That is the underlying reason incrementality belongs to large accounts. It is mathematics rather than tooling.

What is worth testing first

If you can only run one test, run it on whatever looks most suspicious of taking credit for somebody else’s work.

Brand campaigns are always the first candidate. Paying to appear when somebody searches your own name makes sense in some cases, for instance when a competitor bids on your brand, and none in others. A test settles it in weeks.

Retargeting comes second. Chasing people who already added a product to the basket works, but some of them would have returned on their own.

And automated catalogue campaigns complete the podium, for the reason we saw with Advantage+. On Google the closest equivalent is Performance Max.

If you lack the volume for a test

Two middle paths give a reasonable signal without demanding thousands of conversions.

The first is switching a campaign off for a period and watching total business sales rather than platform sales. It is crude and seasonality contaminates it. Even so, when you switch the brand campaign off for two weeks and total sales do not move, you already know something.

The second is asking. A «how did you hear about us» field on the form, left open, gives information no platform will. It is imprecise and still changes decisions.

Neither replaces a control-group test. Both still beat trusting the platform dashboard without checking anything, which is where we start when setting up measurement for a new client.

How the result is read

The result arrives as a percentage lift and, if the test is sound, a confidence interval. That interval is what almost nobody looks at and what matters most.

A result of «+12% with an interval from 2% to 22%» says the campaign adds something, even if the amount is uncertain. A result of «+12% with an interval from −8% to 32%» says nothing: the effect could be negative. The headlines look alike and mean opposite things.

With the percentage in hand, the deciding calculation is simple. Multiply the period’s sales by the incremental percentage, subtract the spend, and see whether margin remains. That number, rather than the platform’s ROAS, decides whether the campaign survives.

The mistakes that invalidate a test

A badly built test produces a number with the same confidence as a sound one, and that is the danger.

1. Stopping early. Checking daily and halting when the result looks favourable produces the result you wanted.
2. Changing something midway. Raising budget, swapping creative or touching targeting breaks the comparison.
3. Running it in an atypical season. Sales periods, Christmas or a campaign on another channel contaminate the result.
4. Measuring the wrong conversion. Optimise to clicks and measure clicks, and the test confirms what you knew while saying nothing about sales. If the sale closes off the website, import it first as an offline conversion.

The first is by far the most common, and it is the same bias that ruins A/B page tests.

A fifth mistake is subtler and specific to this kind of test. If the control group keeps seeing your brand through other channels, the result comes out low. You are measuring that campaign's effect on top of everything else, not the effect of advertising as a whole.

Where to start

Look at how many conversions a month your account produces. Above 500, a formal test on the brand campaign becomes realistic. Below that, start with the crude method: switch that campaign off for two weeks and watch total business sales.

And before building anything, write down what decision each possible result leads to. If the test says the brand campaign adds 10%, do you keep it or cut it? Without that answer in advance, the result gets interpreted to suit whoever is reading it.

In both cases, the number that matters is not in the platform. It is in your invoicing, and that is the comparison we run on any account entering management.

Frequently asked questions