If you'd told me back in college that tests would be fun someday, I'd have chucked the nearest Mountain Dew at you. But recently I've developed a whole new appreciation for the scientific method, and I'm here to testify that subject line split testing is the bee's knees.And it's about to get a lot simpler in Emma. With our upcoming automated split testing feature, you'll be able to easily split a group of subscribers and test different subject lines among them, and then send the version that brings the best results to the remainder of your audience. It'll be just a matter of time until you're increasing open rates and re-energizing your audience base.
Of course, as we anticipate this new feature, it's possible to get ahead of the learning curve and do some manual split tests, too. We recently ran an A/B/C subject line split test in our July Roundup (Delaney wrote all about it here), and last month we tried a simple A/B test in our Agency Insider. Curious how it went down? Read on …
The school of the short subject line vs. the school of the long subject line
We deliver an Agency Insider to Emma's agency partners each month. By comparing mailings over time, we can eyeball every subject line vis à vis that campaign's open rate. We discovered that over the past quarter, three of our campaigns carried detailed subjects, and one bore a short one. And, in fact, the short subject line generated the highest open rate. However, its winning margin of less than one percent elicited the need for an A/B tiebreaker, leading us to our August test. We wanted to know which one our agency partners would respond to more: a quick subject line that grabs attention but doesn't give away much information or a descriptive subject line that could have the last few words hidden in the preview pane?
How we set it up
The process is fairly straightforward. We exported our agency audience, split it in half and imported back to Emma as two groups. Then, we made a copy of the campaign (saved as a new name) and gave each version a subject line:
- Subject line A > Emma Agency Insider: See our split-test results, get tips for tough client conversations and more
- Subject line B > Emma Agency Insider: We've got tests — and answers — to share
I hypothesized that we'd get a winner in B. More and more people read email on mobile devices, and having the entire subject line viewable seemed preferable. I was wrong. The longer headline is winning by a 3% margin so far. (Response metrics continue to collect in real-time, and while the vast majority of openings have happened, we expect a few more to occur over the next couple weeks.) It's not a very dramatic result, nor a wide enough margin to completely alter our subject line strategy, but it surprised me nonetheless.
Next time, we may choose to do a long/short test again, or maybe something different, such as removing the "Agency Insider" title from one version. We won't draw too many conclusions until we're able to test a number of variables — and a number of times.
Making tests work for you
Be a mad scientist of your own, and considering trying a few split tests to learn insights now that you can apply to future tests and future content. Even if your emails gain fairly solid open rates, it's worthwhile to mix things up. Added confidence and competence will be your result, even when your hunch is wrong.
And should you discover surprising results along the way, please share them here. I'd love to hear about the performance of your subject lines — or any other tests you do. We'll be sure to share more news about the split testing feature as it gets closer to release.