Measurement

Holdout Test

An experiment where a control group is excluded from seeing ads to measure the incremental impact of advertising.

A holdout test (also called a ghost ad test) measures incrementality by comparing a group exposed to ads against a control group that doesn't see ads. The difference in conversion rates between the two groups represents the true incremental impact of advertising. For example, you might run a Facebook campaign to 90% of your target audience while holding out 10%. If the exposed group has a 3% conversion rate and the holdout group has a 2% conversion rate, the incremental lift is 1 percentage point, meaning one-third of conversions would have happened anyway without ads. Holdout tests are the gold standard for measuring incrementality, but they require sacrificing potential sales from the holdout group. They're most valuable for always-on channels like retargeting and branded search, where you suspect low incrementality. Many brands discover that 30-50% of conversions attributed to retargeting would have happened anyway.

Related Terms

Frequently Asked Questions

What is a Holdout Test in marketing measurement?

A Holdout Test, also known as a ghost ad test, is a crucial experiment designed to measure the true incremental impact of advertising. It works by comparing the performance of a group exposed to an ad campaign (the test group) against a statistically similar control group that is intentionally excluded from seeing the ads (the holdout group). The core principle is to isolate the effect of the marketing activity. The difference in conversion rates or revenue between the two groups represents the true incremental lift—the conversions that would not have occurred without the advertising. This method is considered the gold standard for moving beyond simple attribution to establish a causal link between ad spend and business outcomes, providing a more accurate view of return on investment (ROI).

How do you implement and calculate the results of a Holdout Test?

To implement a Holdout Test, a marketer first segments the target audience into two groups: an exposed group (e.g., 90% of the audience) and a control group (e.g., 10% holdout). The exposed group sees the ads, while the control group is suppressed from the campaign. After the test period, the incremental lift is calculated by subtracting the conversion rate of the control group from the conversion rate of the exposed group. For instance, if the exposed group converts at 3% and the holdout group converts at 2%, the incremental lift is 1 percentage point. This means that one-third of the conversions attributed to the campaign would have happened organically. Holdout tests are most effective for 'always-on' campaigns like retargeting, where marketers often find a significant portion of attributed conversions are non-incremental.

Why are Holdout Tests more important than traditional attribution models?

Holdout Tests are fundamentally more valuable than traditional attribution models because they measure **causality** (incrementality) rather than just **correlation** (attribution). Attribution models, such as last-click, simply assign credit based on a touchpoint, but they cannot determine if the customer would have converted anyway. A Holdout Test, by using a control group, directly answers the question: 'What would have happened if I hadn't run this campaign?' This distinction is critical for accurate budget allocation. By identifying non-incremental spend, marketers can reallocate budget from campaigns that are merely capturing existing demand to those that genuinely create new demand, leading to a higher true Return on Ad Spend (ROAS).

Want accurate attribution without the complexity?

Causality Engine automates attribution reconciliation and provides real-time insights for Shopify brands.

Join Waitlist →