The GundirGuide to test cell size

Key Takeaways:

• Test cell size matters — You need enough data to trust your results. Larger test cells increase reliability, but the right size depends on your goals, expected response rate, and budget.
• The 200 response rule of thumb — Hitting around 200 responses per cell usually gets you to a 90%+ confidence level — strong enough to make smart, repeatable decisions.
• New to direct mail? — Keep tests simple and cost-effective. A/B splits and cells of 50K-75K can deliver valuable insights without blowing the budget.

The GundirGuide to test cell size

Subscribe to our podcast to get weekly direct mail tips.

Introduction

Every mail plan should incorporate a test. While meeting key performance indicators (KPIs) is always the primary goal, learning from those campaigns is a close second. These learnings help optimize future marketing efforts and improve overall program effectiveness.

To the extent possible, we want to be making data driven decisions. Data driven decisions require us to know if response rates (or other KPI measures) in a test scenario can be trusted to be repeatable. If Test Cell A beats Test Cell B, will it reliably do that again in the future?

That’s where test cell size comes into play. Larger test cells provide more reliable data, but they also cost more. The goal is to strike a balance: reduce the risk of spending marketing dollars on underperforming segments while maximizing the likelihood of drawing conclusive insights.

The key to understanding that tradeoff is statistical significance.

What is statistical significance?

Statistical significance separates random from meaningful results. For example, if we can determine that Test Cell A beats Test Cell B knowing there is only a 5% likelihood that this is a random result, then we can feel pretty confident that it is a repeatable event — that makes it statistically significant. Why? Because that finding implies a 95% confidence level, meaning that if we repeat that test 100 times we are likely to get a similar result 95 times. Not perfect, but pretty close.

How do we determine the appropriate test cell size to test?

There are three key determinants in formulating an appropriate test cell size:

• How much ‘confidence’ are you after? The higher the confidence level, the larger cells need to be. In marketing, we typically shoot for a confidence level in the 90%-95% range.
• What level of response do you expect? The greater the number of responders you anticipate, the smaller test cells can be.
• How big of a difference do you need to detect? The smaller the lift you’re looking for between test cells, the larger your sample size needs to be. If you only care about spotting big differences, you can get away with smaller test cells.

            For example:

The test below is for testing two differing creative approaches to the same data set and reflects the following criteria:

• Test cells of 150,000 each
• Anticipated response rate of 2%
• Seeking a lift of at least 10% between cells

Note: Confidence levels are estimated based on typical assumptions for response rate variance and distribution. Actual significance may vary depending on campaign-specific factors.

Given the criteria above, we can declare the creative in Cell A a winner since it provided a lift in response of over 10% compared to Cell B. This result yields a 99% confidence level. To put it another way, if we ran this exact test 100 times, we are very likely to see the creative in Cell A beat the creative in Cell B 99 times. Most marketers would be happy to roll out Cell A creative to a larger audience given those odds.

See the image below for the confidence level calculations:

Implications for a regular mailer with established controls

Regular mailers will likely be testing against an established “control” mailing — one that has been performing at a predictable level over time. For them, establishing a statistically significant test is somewhat straightforward — they know an acceptable response rate and are likely to have an established lift required to call a winner. Since established mailers are typically mailing at larger quantities they are also likely to have access to data and the resources to pay for a sample size that will yield a 90%+ confidence level.

The dilemma for new mailers

While established mailers have a predictable ROI to help fund testing, new mailers don’t have a response history (or DM revenue stream) to fall back on. In this case, learning is paramount, so we often want to accelerate testing with numerous cells to put us on the path to finding the most promising segments for future testing and rollout. In those cases, we use an A/B split test methodology for each variable being tested and make assumptions to build appropriate test cell sizes to maximize learnings for minimal costs.

With time and testing, mail response will be optimized. But to start, it makes sense to play it conservative. Experience indicates that once we get to about 200 responses we usually reach a confidence level of 90% or better, which is typically strong enough to make informed decisions. Determining the quantity it will take for a new mailer to generate 200 responses is a key part of our discussions with these clients.

Final thoughts

So, to wrap it up, here is a rule of thumb we use for initial testing quantities. We find that test cells of 75,000 are likely to lead us to readable, actionable test results. Dipping down to 50,000 per cell is borderline — meaning you may poke below a 90% confidence level — so some judgement comes into play. The tradeoff for this lower quantity is it addresses budget concerns and gives us the ability to run more test cells in a shorter amount of time.

What works best for your business?

This is a high level primer and it’s not meant to be a how-to guide. So, when it’s time for your direct mail test plan, it’s time to have a discussion with a direct mail data pro. We have a bunch of them here at Gundir and they’d be happy to guide your direct mail journey whether you’re new to the channel or a regular mailer looking for some insights.

Visit our contact page to get started with a direct mail strategy that will support every stage of the customer journey.