Incrementality testing: how to measure what your ads really caused

Incrementality testing measures what your advertising actually caused: the conversions or revenue that would not have happened without a campaign or a budget change. You estimate a baseline of what would have happened anyway, compare it with what really happened, and read the gap with a confidence interval. Attribution reports cannot do this, because they only split credit among the touchpoints a customer saw.

Incrementality is the share of a marketing result that was caused by an ad or a change in spend, measured as the actual result minus a counterfactual baseline of what would have happened without it.

Why isn’t attribution enough to measure ad impact?

Take a hypothetical customer. She watches a video ad on Instagram, which records no click. Three days later she searches your brand name on Google, clicks your brand search ad and buys. A last-click report gives the whole sale to brand search. The obvious follow-up question is the one the report cannot answer: if the brand ad had not been there, would she have clicked your organic listing right below it and bought anyway?

Attribution falls short for three structural reasons:

1. Not every path is visible. Ad blockers, declined cookies, several devices, influencer posts and offline media leave no click trail, so those touches never enter the model.
2. Credit flows to the last touch. Channels that harvest existing demand, such as brand search and retargeting, collect credit. Channels that create demand, such as video and social, look weak.
3. Correlation is not causation. A campaign targeted at people who were already about to buy reports many conversions even if it changed nothing.

The attribution models guide shows how the same sale moves between channels depending on the model. Incrementality testing steps outside that debate. A well-known example: when eBay paused brand keyword ads on Yahoo and MSN in a controlled experiment, almost all of the lost paid clicks moved to eBay’s free organic results, and the researchers found no measurable short-term sales loss (Blake, Nosko & Tadelis). Attribution had credited those ads with every one of those sales.

How does incrementality testing work?

Every causal measurement answers one question: what would we have seen if the ad had not run? That imaginary scenario is the counterfactual baseline. You can never observe it directly, so you build it, either with a control group that did not see the ad or with a model that forecasts the baseline from history. The quality of your measurement is the quality of that baseline.

The two formulas you need are short:

• Incremental lift = actual result − baseline
• Incremental ROAS (iROAS) = incremental revenue ÷ change in spend

Here is a hypothetical worked example. A store raises daily paid social spend from $1,000 to $1,400 for 14 days. The baseline model predicts 312 orders a day without the change; the store actually gets 400.

• Incremental orders per day: 400 − 312 = 88
• Incremental orders over the test: 88 × 14 = 1,232
• Extra spend: $400 × 14 = $5,600
• Incremental cost per order: $5,600 ÷ 1,232 = $4.55
• With a $25 average order value, incremental revenue is 1,232 × $25 = $30,800, so iROAS = $30,800 ÷ $5,600 = 5.5

Note the denominator: the change in spend ($5,600), not the total spend in the window ($19,600). Dividing by total spend would understate the return on the extra budget by about 3.5 times. If you need a refresher on the standard ratio, the ROAS formula guide covers it, including break-even ROAS.

Which incrementality test should you run?

There are three common designs. They differ in how the baseline is built.

Randomized experiments are the gold standard. In a study of large-scale Facebook advertising experiments, common observational methods often failed to recover the effects that the randomized tests measured, sometimes by a wide margin (Gordon et al., 2019). When randomization is not possible, a time-based design with a forecast baseline is the practical option. The best-known method, Google’s CausalImpact, uses a Bayesian time-series model to predict what the outcome would have been without the intervention and compares that forecast with what happened (Brodersen et al., 2015).

A time-based lift test runs in four steps:

1. Record the change: date, channel and type, such as a new channel, a paused campaign or a budget increase.
2. Train on the past: the model learns weekday patterns, trend and calendar effects from at least two weeks before the change.
3. Forecast the baseline: what you would have seen over the next 7, 14 or 28 days without the change.
4. Compare with an interval: actual minus baseline, reported with a confidence interval and the minimum detectable effect.

How do you read a lift test result?

A lift number without an interval is a guess. Two hypothetical results show the difference.

Positive result. The paid social budget increase above produced an average of 400 orders a day against a baseline of 312. The 95% confidence interval for the absolute effect runs from 66 to 108 orders a day, or 21% to 35% relative to the baseline. The whole interval sits above zero, so the effect is statistically significant: the change added roughly 88 orders a day.

Neutral result. A new display network campaign produced 202 orders a day against a baseline of 193. The interval runs from −12 to +30 orders a day, or −6% to +16%. It includes zero, so no effect was detected.

That second sentence is precise for a reason. “No effect detected” is not “no effect”. Every test has a minimum detectable effect (MDE), the smallest lift it can reliably distinguish from noise given your traffic and test length. If this test’s MDE was 18%, all you have learned is that the effect was probably smaller than 18%. Before cutting the campaign, rerun it with a larger change or a longer window. Sizing a test before you run it follows the same logic as sizing an A/B test, covered in the guide to A/B test sample size.

What makes a lift test reliable?

Five rules prevent most bad readings:

1. One change at a time. Keep other channels and campaigns steady during the test, or their effects mix with yours.
2. Make the change big enough. A 10% budget change usually disappears in daily noise. Check the MDE before you start.
3. Keep the pre-period clean. At least 14 days, with no other large change and no tracking outage.
4. Respect the calendar. Holidays, Ramadan and big sale events change buying behavior. A baseline trained across a holiday, or a test window that overlaps one, needs those days modeled.
5. Read the interval, not the midpoint. A neutral result is an invitation to rerun the test, not a verdict to stop spending.

How does marketing mix modeling fit in?

A lift test answers whether one change worked. It cannot tell you how to split next year’s budget across every channel at once. That is the job of marketing mix modeling (MMM).

MMM breaks weekly sales or leads into parts: a base that would happen without advertising, trend, seasonality and holidays, and the contribution of each paid channel. Its inputs are weekly totals of spend and results, so it needs no cookies, clicks or user IDs, and it can see channels that leave no click trail, such as messaging apps and influencers. Open-source models such as Meta’s Robyn and Google’s Meridian use two transformations to describe how media works (Robyn):

• Adstock (carryover): an ad’s effect continues, decaying, into the following weeks.
• Saturation (diminishing returns): each extra unit of spend in a channel returns less than the one before.

Saturation is why the most useful MMM output is the marginal return: the return on the next unit of spend, not the average return so far. A channel with an excellent average ROAS may already sit on the flat part of its curve, while a small channel may still be on the steep part.

MMM outputs usually include a decomposition of results by driver, effect share against spend share, a response curve per channel and a budget planner. Here is a hypothetical comparison of spend share and effect share:

Before trusting a model, check its quality labels: R² (how much of the variation it explains), out-of-sample error on weeks it did not see in training, and whether each channel varied enough in spend for its effect to be separated. Lift tests and MMM also strengthen each other: Meridian, for example, lets you calibrate the model with experiment results (Google Meridian).

Attribution, lift tests or MMM: which should you use?

Each tool answers a question the others cannot. Used together they give a full picture.

A practical cadence: use attribution to optimize campaigns day to day, run a lift test whenever you make a large change, and refit the MMM each quarter to set the channel split.

How do you measure incrementality in AdPix?

AdPix includes campaign lift measurement and marketing mix modeling from the Starter plan up. The Pro plan adds true campaign impact (incrementality) analysis and MMM for each channel’s share of sales. Plans are listed on the AdPix pricing page.

To get a useful first result you need:

• The AdPix tag on your site with your conversion event (purchase or lead) firing.
• Daily ad spend by channel and campaign.
• For a lift test: at least 14 days of stable data before the change, and one clearly dated change.
• For MMM: about 18 months of weekly spend and results, with enough variation in each channel’s spend. A channel whose budget never changed cannot be measured.

Clean channel data comes first. Tag campaign links consistently with the UTM parameters guide, and check where users drop off with funnel analysis before you blame the ads. Then pick the largest budget change you are planning for next month, size it with the A/B test calculator, and run it as your first lift test.

A/B test calculator

Sample size and duration for your next experiment. The free plan covers 100,000 events a month, no credit card. Setup takes under five minutes.

Start freeTry it free