In this article
- Why geography is the right unit for causal advertising measurement
- The four non-negotiables that separate a valid experiment from expensive noise
- What outputs a well-designed geo experiment produces — from iROAS to MMM calibration inputs
- What to do when experiment results and your MMM tell different stories
- The six criteria that make an incrementality estimate trustworthy enough to act on
Controlled experiments are the most rigorous method available for establishing what advertising truly caused. Here is what a geo experiment is, why it works, and what separates a result you can act on from an expensive piece of noise.
The principle behind a geo experiment is ancient and simple: divide your markets, treat some and not others, measure what is different between them. That difference — after controlling for everything else happening at the same time — is what your advertising caused. That is your incrementality number.
The complexity lies in executing it well enough that the number is trustworthy. A poorly designed experiment gives you a confident answer that is wrong. A well-designed one gives you an incrementality estimate precise enough to calibrate your MMM and defend major budget decisions.
Why Geography Is the Right Unit
Geo experiments have been used in advertising measurement since the 1950s, when P&G and Coca-Cola pioneered regional ad campaigns to measure sales lifts. They fell out of fashion when digital marketing made user-level testing possible. They are back now — more important than ever — because the landscape has shifted.
User-level tracking has become unreliable across browsers, devices, and privacy regulations. Geography, by contrast, is stable, non-personal, and inherently isolated. You can run a campaign in Boston and withhold it from Atlanta. No cookie needed. No consent challenge. The separation is physical.
The Principle
Geographic isolation makes geo experiments privacy-safe by design. The separation is physical — you run a campaign in one region and withhold it from another — which means results do not depend on individual-level tracking.
The Four Non-Negotiables of a Valid Geo Experiment
Not every regional analysis qualifies as an experiment. These four conditions define the difference between a rigorous causal estimate and a number that merely looks like one.
An incrementality experiment is not something you run and then analyze retrospectively. Test and control groups must be assigned before any media runs. Post-hoc regional analysis is observation, not experimentation — and it does not establish causality.
Regions are randomly allocated to test and control groups. Randomization controls for confounding factors you did not know existed. Markets are messy. Randomization levels the playing field — it is what separates a rigorous estimate from a biased one.
Small tests are statistically fragile. Too few markets or too short a window and the experiment lacks the power to detect a real effect. Best practice is large-scale randomization: all DMAs in the US, large sets of cities or postal clusters elsewhere. Scale strengthens both internal validity and the ability to generalize results.
If control regions are accidentally exposed to the treatment, or ads do not deliver correctly in test regions, the incrementality estimate is contaminated. A pre-registered analysis plan, live monitoring during the campaign, and confirmed delivery separation are essential.
“A poorly designed experiment gives you a confident answer that is wrong. A well-designed one gives you an incrementality estimate precise enough to calibrate your MMM and defend major budget decisions.”
What a Good Geo Experiment Gives You
A well-executed geo experiment produces four outputs, each serving a distinct role in your measurement workflow.
| Incremental ROAS (iROAS) Revenue generated per dollar of additional spend in test regions versus control, with confidence intervals. This is a direct causal measurement, not an inferred one. | Sales uplift curves Week-by-week trajectory of test versus control, with a counterfactual estimate of what control regions would have sold had they been treated. |
| Response curve shape By testing at different spend levels, you can establish where diminishing returns begin — directly measured rather than statistically estimated. | MMM calibration input Formatted as a Bayesian prior or coefficient constraint, designed to feed directly into the modeling workflow and anchor the model’s estimates in causal reality. |
Why This Matters
iROAS is a direct causal measurement — not an attribution estimate. That distinction is critical when the number is being used to justify a budget reallocation or calibrate a marketing mix model.
When Experiment and Model Disagree
Sometimes an incrementality experiment and your MMM tell different stories. This is not a failure — it is a signal to investigate. The gap often has a reason: the experimental setup may have introduced bias (a contaminated control, non-homogeneous samples), or the experiment may be measuring a short-term window that does not capture the carryover effects the model accounts for.
One useful approach when results diverge: treat the experiment as an upper bound of the incremental uplift. The experiment shows what is possible under controlled conditions; the model shows what is plausible for planning, given the full market context. Used together as a triangulation rather than a competition, both outputs become more reliable over time.
The Rule
When experiment and model disagree, investigate before choosing sides. A short-term experiment may not capture carryover effects; a model without calibration may overstate them. Used together as triangulation, both outputs improve over time.
What Makes a Geo Experiments Incrementality Estimate Trustworthy: Key Takeaways
- ✓ Random assignment of regions — not convenience sampling
- ✓ Designed and pre-registered before media runs
- ✓ Sufficient scale and duration to achieve statistical power
- ✓ Clean delivery separation confirmed during the campaign
- ✓ KPIs and geography aligned with the MMM to enable direct calibration
- ✓ Carryover effects understood and accounted for when comparing with the model
Frequently Asked Questions
What is a geo experiment?
A geo experiment divides markets into test and control groups, runs advertising in the test group while keeping the control group unexposed, and measures the difference in outcomes between them. That difference — after controlling for what else was happening at the same time — is the causal impact of the advertising.
Why use geography rather than individual users as the experimental unit?
Geographic separation is physical and does not require individual-level tracking. With user-level tracking becoming unreliable due to privacy regulations and cookie deprecation, geography offers a stable, privacy-safe alternative that still produces statistically robust results.
What happens if you analyze results after the campaign has already run?
Post-hoc regional analysis is observation, not experimentation. Without pre-registered test and control assignments, you cannot rule out the confounding factors that a randomized design would have controlled for. The result is a correlation, not a causal estimate.
What does a geo experiment produce that an MMM cannot?
A geo experiment produces a direct causal measurement — iROAS, sales uplift curves, and response curve shape — derived from an actual control group rather than statistical inference. That makes it the gold standard input for calibrating an MMM: it anchors the model’s estimates in observed reality.
What should you do when an experiment and your MMM give different results?
Treat it as a signal to investigate, not a conflict to resolve by choosing one over the other. The experiment may be capturing a short-term window that misses carryover effects; the model may be overestimating them without calibration. Used together as triangulation, both outputs improve over time.
Related Articles
| 📊 Guide Incrementality Measurement: The Complete Series Overview | 🎯 Solution Incrementality Measurement at MASS Analytics | 📈 Solution Marketing Mix Modeling Solutions |
Ready to measure what your advertising actually causes?
MASS Analytics helps you design, run, and calibrate geo experiments that feed directly into your MMM — so every budget decision is grounded in causal evidence.
Explore Incrementality Solutions Book a Demo