Incrementality testing in marketing is a method used to measure the true impact of a marketing campaign or channel by determining the incremental value it adds beyond what would happen naturally or through other efforts. This approach helps you understand whether your campaigns drive additional outcomes (e.g., sales, leads) or simply take credit for results that would have occurred anyway.
Incrementality testing is currently in Beta. We are accepting new clients to set up a geo-test. Contact our team via support@billygrace.com to set up an incrementality test.
How does it work?
A geo-experiment to explore the incremental impact of a channel on the number of sessions.
We turn off a channel (for example, Meta) in a specific region and observe the divergence between the on/off regions to gain insights into the incremental effects.
The KPI to monitor will be website traffic/sessions.
It is not an A/B test. The channel will be off in the tested region and on in the rest of the country.
An example of an Incrementality test:
Let’s say you’re running Meta ads. To measure the incremental value of your Meta Ads, you decide to conduct a geo-holdout test:
Control Region: except for the holdout region, in the rest of the country, you continue running Meta ads as usual.
Holdout Region: In Noord-Holland, you pause all Meta ads for the duration of the test (or just exclude Noord-Holland from targeting).
For several weeks, Billy Grace tracks and compares user behavior in both regions. By comparing the data, Billy Grace can determine how much of the website traffic in Noord-Holland was lost due to no Meta Ads, while the rest of the country serves as a baseline where ads were running.
Steps to set up the test:
Feasibility check
Billy runs simulations on historical data to identify the most suitable regions and channels for testing and determine the necessary time window (e.g., 15 days or 1 month).
Run the experiment for 15 days or 1 month by turning off advertising in the selected region.
Calibrate UMM results based on outcomes.
Result:
The experiment results will be used to calibrate the UMM models, increasing the certainty in the estimates —> Validation of the estimated UMM view effects. This gives insight in the incrementality of your advertising.
Incrementality testing is interesting when you have the following questions:
Are my campaigns driving results?
Problem: Are the observed outcomes (e.g., sales, clicks, sign-ups) due to the campaign, or would they have happened organically without it?
Insight: Incrementality testing helps determine the causal impact of the campaign, ensuring resources aren't wasted on initiatives that merely capture existing demand.
Which marketing channels deliver the most value?
Problem: With multiple marketing channels (e.g., social media, email, paid search), it's unclear which is truly contributing to incremental growth.
Insight: By isolating the impact of each channel, you can identify high-performing tactics and reallocate budgets to maximize ROI.
What is the optimal budget allocation?
Problem: How should the marketing budget be distributed across campaigns or channels to achieve the best return on investment?
Insight: Incrementality testing reveals diminishing returns on oversaturated channels and highlights underfunded areas with untapped potential.
Incrementality testing is currently in Beta. We are accepting new clients to set up a geo-test. Contact our team via support@billygrace.com to to set up an incrementality test.