Control groups in marketing: how to measure what works
A control group in marketing is the set of customers you leave out on purpose, to see what a campaign really caused. Holdout vs control groups, sizing, setup.
Table of contents
- Key takeaways
- What a control group is in marketing
- Why before-and-after cannot prove a campaign worked
- Holdout group vs control group
- Five ways to build a control group without A/B tools
- How to set up a marketing control group
- What quietly breaks a control group
- When a control group is not the answer
- Where to start
- FAQ
The retention campaign went live in January: check-in emails and a loyalty offer for customers who had contacted support. By April, churn in that group was down. The marketing team put the two facts on one slide, and for about a week it was the best slide in the company. Then someone from finance pointed out that a price increase had been withdrawn in February, and someone from sales mentioned that the main competitor had spent March apologizing for an outage. The slide was still true. It just no longer proved anything, because the campaign had no control group.
A control group in marketing is a set of customers who qualify for a campaign but are deliberately left out of it, ideally at random, so that what happens to them shows what would have happened anyway. The difference in results between the customers who got the campaign and the control group is the part the campaign can claim, usually called incremental lift.
Most marketing and experience teams are in that position when they try to claim a result: they know what changed and what happened afterward, and nothing connects the two. The honest connection is a comparison with customers who did not get the change, whether it is an email, an offer or a new complaint process.
Key takeaways
- A control group in marketing is a comparable set of customers left out of a campaign on purpose, so the difference between them and everyone else is the campaign’s real effect.
- A before-and-after comparison cannot say why a number moved, because everything else changed over the same time; a control group holds time constant.
- “Holdout group” and “control group” are often used interchangeably; where they differ, a holdout usually means customers kept out of a whole program for longer.
- You do not need a testing platform: a branch, a region, a cohort, a staggered rollout or a random slice of customers can each serve as the control.
- The four things that ruin a control group are contamination, groups too small to show the effect, letting the treatment group choose itself, and hunting for whichever metric moved.
- Never hold out safety fixes, refunds or corrections of genuine errors; when an improvement is held back, stagger it where you can and keep the wait brief.
What a control group is in marketing
In direct marketing, a control group on a campaign is routine: a random slice of the mailing list does not get the offer, and the lift is the difference in purchase rate between the two groups. The number most campaign reports show instead, how many recipients bought, is not the effect. Some of those customers would have bought anyway, and only the control group says how many.
The same logic works for changes that are not campaigns at all, such as a new complaint-handling process or a rewritten onboarding sequence.
Three things are often mistaken for a control group and are not. A before-and-after comparison has no control, only a different time. A segment comparison (customers who opened the email versus those who did not) compares people who differ for reasons unrelated to the campaign. And a survey sample is a way of observing customers, not a way of assigning who gets what. A control group exists only when the people running the change decided, in advance, who would not get it and why those people are comparable.
Why before-and-after cannot prove a campaign worked
Comparing this quarter to last quarter tells you what happened over time. It cannot tell you why, because everything else changed over the same time: the season, the economy, the marketing calendar, the product, the competitor. Any of them could explain the difference, and someone in the room always finds one.
A control group removes that problem by holding time constant. Two groups of customers live through the same quarter, the same season, the same competitor outage. One gets the campaign and one does not, so the difference between them at the end belongs to the campaign.
The usual objection is that some changes cannot be A/B tested: you cannot serve a new store layout or a retrained call center team to half the people who walk through the door. That is true of the tools, not of the method, and the role of measurement in keeping a program alive depends on using it anyway.
Holdout group vs control group
The two terms are often used interchangeably, and within a single test the difference is mostly emphasis: “holdout” stresses the act of holding customers back from something that is otherwise going to everyone. Where people do draw a line, it usually runs like this.
A control group is the untreated comparison group in a specific test: the customers who did not get this email, offer or process, measured over the same period as those who did. When the test ends, they usually get whatever won.
A holdout group usually means customers deliberately kept out of a campaign or a whole program, often for longer and sometimes across many campaigns, to measure the incremental lift of the marketing as a whole. A retailer might keep a small random slice of its loyalty members out of all promotional email for a year, for example, to see how much the email program adds. Some teams call this a universal holdout or, confusingly, a global control group, which shows how loosely the words travel.
| Control group | Holdout group, where distinguished | |
|---|---|---|
| Usual scope | One test or campaign | A program, a channel or many campaigns |
| Usual duration | The length of the test | Often longer, sometimes ongoing |
| Question it answers | Did this change work? | How much does this program add overall? |
Long-running holdouts carry two costs a single test does not. They give up whatever the marketing would have earned from the held-out customers, which is one reason they tend to be a small slice. And they drift, as excluded customers hear about offers through other channels or stop being comparable, so they need checking and occasional refreshing.
Nobody enforces the distinction, so when a colleague says “holdout”, ask two questions: held out of what, and for how long?
Five ways to build a control group without A/B tools
No testing platform is required, only a way to split customers into two comparable sets and give one of them the change first.
A branch or a store. Roll the change out in some locations and not others, matched on size, customer mix and last year’s results.
A region. The same idea at a larger scale, which marketers often call a geo test.
A cohort. Give the change to customers who joined after a set date and compare them with those who joined just before, which suits onboarding and the leading indicators of lifetime value that show up in the first ninety days.
A staggered rollout. Each wave that has not started yet is the control group for the waves that have, and nobody is denied anything, only scheduled.
A random ten percent. For anything you control at the customer level, such as an email, an offer or a follow-up call after a complaint, hold a random slice on the old treatment for a set period. Random matters: choosing the slice by hand brings back the bias you are trying to remove.
| Holdout unit | Best for | Main risk |
|---|---|---|
| Branch or store | Process and training changes delivered in person | Staff in holdout branches copy the new process informally |
| Region | Regional campaigns and policy changes | Local events, such as weather or a new competitor, skew the result |
| Cohort by join date | Onboarding and early-relationship changes | Seasonality in who joins when |
| Staggered waves | Anything rolling out everywhere anyway | Later waves hear about the change and expect it |
| Random customer slice | Emails, offers and follow-ups | Exclusion errors in sends and scripts |
How to set up a marketing control group
You can do this next quarter without a new tool.
- Write down the outcome and the timeframe before anything starts. “Purchase rate within 90 days among eligible customers” is an outcome. “Better engagement” is a hope. Use the same definitions your retention reporting uses, so the result can be compared with the numbers everyone already sees.
- Choose the unit of holdout: customer, branch, cohort or wave. Pick the smallest unit you can actually keep separate.
- Decide how big the control group must be. Ask: is the difference I am hoping for bigger than the difference I would see between two random groups by chance? If nobody can answer that, ask someone who can before you start, or make the control group larger than feels comfortable.
- Assign at random, or as close as you can get. For branches or regions, match on last year’s numbers and flip a coin for which side gets the change.
- Protect the boundary. Tell the people who need to know, exclude the control group from every related send and script, and check monthly that it is still clean.
- Compare only what you wrote down in step one. Resist the urge to hunt for some metric that moved. Something always moved.
A worked example (illustrative, round numbers)
A retailer wants to know whether a win-back email series brings lapsed customers back. It has 20,000 lapsed customers, holds out a random 2,000 and sends the series to the other 18,000. After ninety days, 9 percent of the emailed group has made a purchase, which is 1,620 customers. In the control group, 6 percent has, without any email at all.
The campaign report would say the series won back 1,620 customers. The control group says most of them were coming back anyway: the lift is three points, which on 18,000 customers is roughly 540 people the series can claim. Whether 540 returning customers justify the cost is a separate question, and it is now a question with an honest number in it.
What quietly breaks a control group
The method is simple. The execution has four traps, and I have fallen into each of them.
Contamination. The control group finds out, or gets treated by accident: the holdout branch copies the new process, or someone forgets to exclude the control group from a send. Once the two groups have been treated the same, the comparison is dead, so guard the boundary throughout, not just at the start.
Groups too small to see anything. If the effect you are hoping for is a couple of points of retention and your control group is 80 customers, you will not see it. A group that is too small does not just fail to prove the change worked; it produces a random result that someone will present as if it meant something.
Comparing volunteers with non-volunteers. Customers who opted into the pilot differ from those who did not, in ways that have nothing to do with the pilot. So do branches whose managers put their hands up. If the treatment group chose itself, you have measured enthusiasm. This is a recurring problem in pilot programs, and building buy-in for a pilot has to be done without letting the enthusiasts pick the sites.
Hunting for the metric that moved. If the outcome you wrote down did not move, the temptation is to look at twelve others until one did. With twelve metrics, there is a fair chance one moved by luck alone. Report the outcome you wrote down first, and label anything else as exploratory.
When a control group is not the answer
Sometimes holding out is wrong, and sometimes it is impossible.
Never hold out safety, refunds or the fixing of a genuine error. Those go to everyone, immediately, and you find another way to measure. A brief, bounded holdout of an improvement is different: it buys the evidence to keep the improvement funded once the initial excitement fades, the fate of most pilots that cannot prove themselves (Corporate attention deficit and why pilot programs fail is about that fate).
When the change cannot be split. A rebrand, a new pricing structure, a company-wide policy that customers would compare across the boundary. Here the fallback is a before-and-after with the other explanations written down in advance, before the results arrive.
When the customer base is too small. A business with a few hundred customers cannot split them and expect to see a modest effect. Use customer-level leading indicators instead, and be honest that the evidence is suggestive.
A control group also tells you whether a change worked, not why. The why lives in what customers in both groups said, which is the case for reading qualitative and quantitative feedback side by side.
A control group will not make a weak campaign look strong. That is the point. What it does is let a strong one look strong to the people who decide whether it continues.
Where to start
- Pick the next campaign or change that is going to more than one place or segment. An email series, an offer, a script, a policy. Rollouts that are already planned in phases are the easiest.
- Write the outcome, the definition and the timeframe on one page. Get the person who owns the customer analysis to agree the definition before anything starts.
- Choose which wave, branch or slice is the control group, and make it random or matched. Treat that group as the most important one in the plan.
- List every send, script and system that would need to exclude it. Do the exclusions, then check them at the end of the first month.
- Put a date in the calendar for the comparison. One number, the one you wrote down, for each group, on one slide.
FAQ
What is a control group in marketing?
A control group in marketing is a set of customers who qualify for a campaign but are deliberately left out of it, ideally chosen at random. Because they live through the same period as the customers who got the campaign, the difference in results between the two groups shows what the campaign caused rather than what would have happened anyway. Without one, a report shows who responded, not who responded because of the campaign.
What is the difference between a holdout group and a control group?
The terms are often used interchangeably. Where people distinguish them, a control group is the untreated comparison group in one specific test, while a holdout group usually means customers kept out of a campaign or a whole program, often for longer or across many campaigns, to measure overall incremental lift. When someone uses either term, it is worth asking what exactly is being withheld and for how long.
How big should a control group be?
Big enough that the effect you expect is larger than the difference two random groups would show by chance. That depends on the size of the effect, the baseline rate of the outcome and how much it varies, so there is no single right percentage. The smaller of the two groups limits precision, and detecting an effect half as large takes roughly four times as many customers. A standard sample-size calculation before the test gives the number.
How do you calculate lift from a control group?
Subtract the control group’s rate from the treated group’s rate for the outcome you chose in advance; the difference is the lift. Multiply it by the number of customers who got the campaign to estimate how many results the campaign caused. Some teams express lift as a percentage of the control group’s rate instead, so say which one you mean.
How long should a holdout last?
Long enough for the outcome you wrote down to be observable, and no longer. For a campaign measured by ninety-day purchase or retention, that is about a quarter. Program-wide holdouts can run much longer, but they need checking for contamination and drift, so measure an early indicator where you can and let the lagging outcome confirm it later.
