New to incrementality testing? Read our Incrementality Testing article to understand how it works before setting up your first test.
Incrementality Testing is available as a paid feature. Interested in getting started? Contact our team via support@billygrace.com.
Feasibility check
Billy Grace runs a feasibility check on your account in the background every two weeks, so you don't need to request one. The check confirms you have enough data to measure incrementality reliably, and it identifies which channels and countries can be tested.
Once your account has passed and you have purchased a test, the Create test button is active. The steps that follow then show only the channels and countries you're eligible to test, so everything available to select is ready to go.
If the Create test button is greyed out, check whether a test is already running. Test creation is paused while a test is in progress because you can only run one test at a time. Otherwise, either your account has passed the check but you haven't purchased a test yet, or your account hasn't passed the check.
A few things can hold up a pass:
Not enough data. An account needs at least three months of data before a test can be measured properly.
A low number of daily conversions. Incrementality tests compare a holdout group against the rest of your activity, so there need to be enough conversions each day for a difference between the two to be measurable.
Low spend on a channel. When spend is small, the attributed value of that channel is small too, which leaves too little signal to measure a reliable effect.
Recent promotions or other one-off activity. These make it harder to separate the effect of a test from everything else happening at the same time.
One thing the feasibility check can't see
Passing the feasibility check means your historical data supports a test, it doesn't mean nothing else is about to change. The check has no way to know about a campaign overhaul, a new website, or a big strategy shift you already have planned for the coming month.
If you're planning changes like that for the channel or region you're about to test, we'd recommend holding off until after those changes have settled in, or choosing a different channel or region for now. Otherwise, you may not be able to tell whether your results reflect the test or the other change(s).
Not sure whether an upcoming change would affect your test? Reach out before you launch, we're happy to help you time it right.
Contact support@billygrace.com for more information.
Set up the test
You can find Incrementality Tests in the left-hand menu. Click Create test to open the setup page.
Give your test a clear name to track it easily.
Select channel & goal
Incrementality tests can currently be run on Meta and Google. Optionally, you can also specify a cluster within a channel. When you select a cluster, you can click to see which campaigns are included. These are the campaigns that will be paused in the test region during the experiment.
Channel: The marketing channel whose ads will be paused in the holdout region during the test. If you select only a channel, all campaigns within this channel will be paused.
Cluster of channel: A specific group of campaigns within the selected channel. Selecting a cluster limits the test to those campaigns only, rather than pausing the entire channel in the test region.
Goal: The primary metric Billy Grace monitors during the experiment. For Google and other click-focused channels, you should select conversions. For Meta and other upper-funnel channels, you should select sessions.
Select test region
Choose the region that the test will be conducted. The chosen region will need to have its budget set to 0 so the rest of the regions can provide more insights.
Country: The country in which the experiment runs.
Region: The specific region where the selected channel's budget will be set to 0 for the duration of the test.
Test schedule & ad account
The test runs for one month from the selected start date. We may recommend extending it based on its performance. You can run one incrementality test at a time.
How to start the test:
In your ad account, set the budget for the selected channel and test region to 0. Do this within 7 days of clicking Start test.
Once the spend reaches 0, the test starts automatically. This is detected during the next data refresh (within 24 hours).
Important: If any spend is detected in the test region during the duration of the test, the test will fail.
Improve my UMM model
By allowing Billy Grace to use the test results, your UMM model will be trained with the outcomes of the completed experiment. For example, if the test shows that Meta's impact on sessions was higher or lower than UMM predicted, those findings feed back into the model and recalibrate the estimates. The result is a more accurate attribution for the tested channel going forward.
You are all set. Check this article to learn how to interpret test results.
Note: We recommend waiting at least one month before launching a new test. This cooldown period helps reduce the impact of the previous test, giving you cleaner and more trustworthy results.