The Why, What, When, And How Of Direct Mail Creative A/B Testing

Key Takeaways on A/B testing:

• A/B testing is often the most viable testing technique.
• When planning your A/B tests, be clear on what single variable you want to test in each segment.
• Not all tests need to be—or should be—A/B tests.
• It’s only effective when isolating a single variable to test in the creative.

Listen to the podcast: Addressed for Success

The Why, What, When, and How of Direct Mail Creative A/B Testing

When it comes to improving the performance of direct mail creative, A/B testing is often the most viable testing technique. The why is simple: it’s because mail quantities are seldom large enough to get a read on multiple test variables between creative packages. 

Now, what exactly is A/B testing? An A/B creative test is a two cell test that compares one variable—and one variable only—across two creative executions. An A/B test assumes that all control and test versions are being mailed to the exact same target. And oftentimes a mailing will have a number of A/B tests running in one drop. A simple example is testing two elements versus the control in separate cells. If the control is always “A,” then A/B might be a different offer vs. the control and A/C might be a different envelope vs. the control.

When planning your A/B tests, be clear on what single variable you want to test in each segment, and then keep everything else between the two packages as similar as possible. For example, an A/B letter design test in a #10 package means only changing the letter design. To get a clean read on test results, strive to keep the envelope, offer, message, and any other direct mail components the same. 

Here’s A Sample Scenario: Tip-On Card Vs. Check Letter Format

A loan originator has a control package that uses a #10 envelope with an 8.5” x 11” letter, and a tip-on card. The agency recommends testing a faux check-letter in place of the tip-on card. To make that work, either the letter size needs to change to 8.5” x 14” to add room for the faux check and keep the rest of the creative the same, OR the copy and design need to be modified to keep the letter the at 8.5” x 11” and make room for the faux check. In this example, experience has taught us that a longer letter size is probably not going to materially influence response as much as modifying content and design could. But if adding the faux check changed the envelope size—or was accompanied by a new call-to-action—test results would be less clear on what is really driving the change in response.

Format testing can be challenging with regards to A/B testing. No one would expect the contents of a #10 letter to translate to a 6” x 9” postcard, for example. In these cases, strive to communicate the same value proposition, maintain the letter’s messaging structure, and translate the key design elements into the new format. 

When to use an A/B test scenario: 

A/B testing is an incremental approach to improving mail performance. Not all tests need to be—or should be—A/B tests. Beat the control tests are about testing big new ideas — creative concepts, new formats, fresh design, and more. New product introductions deserve broader testing as well, since there is little-to-no marketing history for reference. In general, if you are looking for “breakthrough” performance, an A/B variation is not likely to get you there. Once a new control is established, however, A/B testing is your road to ongoing optimization.

First-time mailers of established brands require a thoughtful approach. If there is an established message that translates well into direct mail, then the best path may be to run that message with an A/B offer, or A/B design test. In other cases, there may be more of a need for overall package design and content exploration, similar to a new product intro scenario. 

The bottom line on A/B testing:

It’s only effective when isolating a single variable to test in the creative. It’s appropriate when controls are in place, and your goal is constant, incremental optimization of existing package performance. Finally, use judgment to minimize the impact on test readability when creative compromises are required .

Have questions about adding direct mail to your marketing mix? Drop us a line.

 Did you enjoy this content? Sign up for our newsletter.