Exploring an A/B subject line split test

Setting up a split test is easy -- and about to get easier

If you'd told me back in college that tests would be fun someday, I'd have chucked the nearest Mountain Dew at you. But recently I've developed a whole new appreciation for the scientific method, and I'm here to testify that subject line split testing is the bee's knees.

Emma Agency Insider
Click to view August's Agency Insider.
And it's about to get a lot simpler in Emma. With our upcoming automated split testing feature, you'll be able to easily split a group of subscribers and test different subject lines among them, and then send the version that brings the best results to the remainder of your audience. It'll be just a matter of time until you're increasing open rates and re-energizing your audience base.

Of course, as we anticipate this new feature, it's possible to get ahead of the learning curve and do some manual split tests, too. We recently ran an A/B/C subject line split test in our July Roundup (Delaney wrote all about it here), and last month we tried a simple A/B test in our Agency Insider. Curious how it went down? Read on …

The school of the short subject line vs. the school of the long subject line

We deliver an Agency Insider to Emma's agency partners each month. By comparing mailings over time, we can eyeball every subject line vis à vis that campaign's open rate. We discovered that over the past quarter, three of our campaigns carried detailed subjects, and one bore a short one. And, in fact, the short subject line generated the highest open rate. However, its winning margin of less than one percent elicited the need for an A/B tiebreaker, leading us to our August test. We wanted to know which one our agency partners would respond to more: a quick subject line that grabs attention but doesn't give away much information or a descriptive subject line that could have the last few words hidden in the preview pane?

How we set it up

The process is fairly straightforward. We exported our agency audience, split it in half and imported back to Emma as two groups. Then, we made a copy of the campaign (saved as a new name) and gave each version a subject line:

  • Subject line A > Emma Agency Insider: See our split-test results, get tips for tough client conversations and more
  • Subject line B > Emma Agency Insider: We've got tests — and answers — to share

I hypothesized that we'd get a winner in B. More and more people read email on mobile devices, and having the entire subject line viewable seemed preferable. I was wrong. The longer headline is winning by a 3% margin so far. (Response metrics continue to collect in real-time, and while the vast majority of openings have happened, we expect a few more to occur over the next couple weeks.) It's not a very dramatic result, nor a wide enough margin to completely alter our subject line strategy, but it surprised me nonetheless.

Next time, we may choose to do a long/short test again, or maybe something different, such as removing the "Agency Insider" title from one version. We won't draw too many conclusions until we're able to test a number of variables — and a number of times.

Making tests work for you

Be a mad scientist of your own, and considering trying a few split tests to learn insights now that you can apply to future tests and future content. Even if your emails gain fairly solid open rates, it's worthwhile to mix things up. Added confidence and competence will be your result, even when your hunch is wrong.

And should you discover surprising results along the way, please share them here. I'd love to hear about the performance of your subject lines — or any other tests you do. We'll be sure to share more news about the split testing feature as it gets closer to release.


Learn more about becoming an Emma agency. Inquire here.

The Brainiac Guide to Welcome Email Automation

Advanced email analytics with Litmus

Using Litmus and Emma to preview your email in various email clients and study engagement

Benjamin Franklin once said that the only certainties in life are death, taxes and the fact that various email programs display HTML differently. Well, he may not have been familiar with that last one. But as email marketers know, it's a truth that adds a level of difficulty to designing for email.

Thankfully, we've got tools for that kind of thing. Litmus is a program that gives marketers a firsthand look at how newsletters render across the major email programs, and it also shows which ones your recipients are using. Last month, Emma took a Litmus for different kind of test drive. We've been using the system for quite a while now, but trying some of their more advanced features this time around gave us some fascinating insights.

Let me introduce two of the Litmus features that we found useful, as well as the results from our own newsletters.

Email previews

Because of the plethora of email clients out there, making campaigns look good everywhere is an uphill battle. Emma's designers are stars at making your stationery display consistently, but once you add images and text to your campaign, you can bet that it won't look exactly the same. And don't even get me started on Outlook. (Here's an example of Emma's old newsletter in Outlook 2007.) To make matters more complicated, email clients span across three environments: desktop software (like Outlook and MacMail), web software (like Hotmail and Gmail) and mobile. For the purposes of this post, that's all you need to know. But if you're curious about rendering engines, which actually perform the task of displaying HTML, you can learn more here.

With a basic Litmus account, you send your email to a test Litmus address to see how your email looks on all major email clients in an instant. From there, you can browse through the clients, scroll on the mobile phones and even turn preview panes on and off to see all preference configurations. It really takes the guesswork out of it.

If you decide to go with a plus or premium account, you'll actually see what emails clients are represented in your audience, and by what percentages. With this data, you can get a sense of just how mobile your subscribers are and how much your campaigns are affected by Outlook's quirks.

Litmus allows you to preview campaigns in all major email clients.


If you're using Emma, you're already getting a good idea of your reader engagement through the response section. Litmus gives you even deeper analytics, at their plus and premium levels. The report tells you exactly how many seconds your audience spends with your emails and categorizes the whole group into "read," "skimmed" and "glanced or deleted." It's even organized by email client.

The results: Litmus in action

We used Litmus for two of Emma's newsletters, our August Roundup (a newsletter sent to our entire community) and our Agency Insider (sent to our agency partners). (To subscribe to either or both of these, go right ahead here.)

Litmus' email previews allowed us to test our campaigns before their send-offs. Then after sending, we dove into the engagement and email client details. We learned a few things along the way, including…

  • The audiences for our community-wide newsletter and agency-specific newsletter are not that different. At over 80% for each test, desktop email clients are still king. Our general community has a higher percentage of Outlook users, while our agencies prefer Mac Mail; those were the #1 and #2 email clients for both.
  • Our mobile readers, despite being a significant minority, were extremely engaged. Over half of mobile recipients fell into the "read" category, spending the most time with our emails. Maybe it simply takes longer to read and digest an email on mobile. Or, maybe folks who make time to check email on-the-go really want to receive the message.
  • Our readers are environmentally friendly. Fewer than 10 readers chose to print out the newsletter.

Pretty interesting, right? You may find that you know your audience better than you expect — perhaps your assumptions are right on the money. Or, you may find that more readers than you realize are using mobile devices and that your mobile strategy needs a tune-up.

Even if the results don't lead to major changes right away — we're pretty pleased with how Emma's data stacked up, for example — it's useful to document the data as a benchmark. Gradual changes to your reports over time will indicate an evolving audience, and it'll allow you to keep your content and formatting fresh. Got anything interesting to share about your own email testing? Please share any insights in the comments. We'd love to hear about it.

And for the record, we don't have any special relationship with Litmus — we just think it's a handy tool, so we wanted to share it with you.

Emma’s in-house “Bike to Jack and Back” auction

A personal connection with multiple sclerosis leads to an innovative fundraising idea

It's Emma's second year participating in Jack Daniel's "Bike to Jack and Back" MS Ride, and our team of riders is ready to kick (nay, pedal) some serious tail on October 1st and 2nd. Kelli Liszka and Jamie Bradley share their stories of how we got involved in this cause. Plus, read on for tips about a fun fundraising idea that any company can do — for any cause.


Kelli Liszka
by Kelli Liszka

Every year, the National MS Society teams up with Jack Daniel's for a bike ride called Jack and Back. Riders start just south of Nashville and ride to Lynchburg, Tennessee, where all of the world's Jack Daniels whiskey is made, and the next day everyone rides back to the start. It's a cool 150-mile roundtrip.

Emma's involvement with Jack and Back started when a friend told me about the event in 2010. The MS Society was having a sale on ride registrations, and I sent it around the Emma house to see if anyone would be up for it. There was a pretty good response, and the first Jack and Back Team Emma was born.

This year our team has grown from 11 to 17 riders, consisting of Emma staffers, friends and family members. A group of Emma volunteers will also help out at the finish line, setting up camp and cheering on the hundreds of riders. We've been hitting the roads to get our legs ready for the ride and are hoping the weather will be in our favor this year.

Jaime Bradley
by Jamie Bradley

In April of last year, I lost my mother to multiple sclerosis. I know, that's kind of a bummer-y way to start my portion of this post, but stick with me — there's a happy ending. I realized pretty quickly that there are only so many Lifetime channel marathons and long phone calls home one can stomach before thinking, "I should probably do something — anything — other than this."

Along came Kelli's email about Jack and Back. I don't own a bicycle, but that didn't stop me from participating. I brainstormed with some other not-so-athletic folks at Emma about ways that the entire office could get involved.

We decided to hold an online auction for all of our staffers to raise money and have fun. Our auction is in its second year, and it's a huge hit. Even people who don't want to bike 150 miles or volunteer during the race love the idea of a fierce bidding war for homemade cheesecake. I asked for all willing participants to tell us a "safe-for-work talent, good or service" that they'd like to auction off, and cheesecake was just the start. We got quite the array of offerings, from Doubles Tennis with CEO Clint Smith to A Deluxe Car Tune-up by designer Seth Wood to A Full Night of DJing For Your Next Dance Party by developer Josh Mock.

We posted all of the offerings online, and on auction day, staffers bid for their favorite items. The bidding gets a little ruthless, I assure you. Last year, we raised a little more than $1,300. This year? $2,575 and some change!

I'm proud it's become an Emma tradition; our donations are making a huge difference to other families like mine.


Want to pull off a company-wide silent auction of your own? Here are a few quick tips:

  1. Create a survey to solicit participation. Encourage coworkers to submit anything from baked goods to singing lessons to a night of babysitting. Take a peek at our survey here.
  2. Export the survey results from Emma's response page to an Excel spreadsheet. Now, it's time to post entries in a spot where your officemates can easily view and bid on them. We used a platform called Jive, but you could use your company's internal blog, build a simple landing page or even coordinate bids via email.
  3. Drum up excitement and explain the rules. Check out the email campaign that Kelli and Jamie sent to the Emma staff. (We've modified the email just a bit so the links are no longer live.)
  4. As bidding time ticks away, build momentum with another email campaign. See ours here.
  5. Finally, once the auction has closed, share news of the winners and money raised. It's super exciting.

That's all there is to it. It's pretty simple, lots of fun and gets everyone involved. Have you got any company fundraising ideas up your sleeve? We'd love to hear them.

Emma takes you back to school

A roundup of school-related email campaigns

The smell of dry erase markers and school bus exhaust. Suddenly re-acclimating the entire family to 6:00 am morning alarms. Fall leaf motifs now that it's finally cooler than 90 degrees in the shade. Yes, for the students among us, summer is officially over, and school is back in session.

Luckily, a lot of schools and businesses help parents and students get back in the loop by sending regular email communications in preparation of the first day of school — and all through the school year.

Take a gander at some campaign examples that get an A+ from us …

Percy Priest Elementary

Percy Priest Elementary sends a weekly newsletter to parents to keep them up-to-date on everything from where to buy official Percy Priest Elementary Tigers gear to quick links for important information on bus routes and lunch menus. Also, I love that the focal point is a personal note from the school principal. In the end, they're delivering timely and relevant information that's also visually appealing. Oh, and they have a knitting club? If only I could go back to school (and back in time) …

+ View the online version of the campaign
+ Visit Percy Priest on Facebook

The ScrapKins

The ScrapKins is a Brooklyn-based organization that aims to get children excited about recycling and teaches them fun ways to incorporate social consciousness via cute-as-can-be little monsters. This particular email is announcing a new game for iPhones that is both educational and, well, pretty fun too. So, as your little ones in the backseat get restless in the carpool lane, they can learn some valuable life lessons while you jam out to your favorite Jon Secada compilation CD.

+ View the online version of the campaign
+ Visit their blog

The City of Orange City

Schools aren't the only Emma customers getting in on the back-to-school spirit. This example from one of my favorite Emma customers, The City of Orange City, IA, is chock-full of great goings-on, but specifically seeks to highlight some kid-friendly activities around town. Recorder Lessons for only 50 cents? Preschool Story Hour at the library? Don't mind if we do, Orange City. And it's lovely to see that they've incorporated social media links at the bottom — it's a great way to keep their audience clicking for more.

+ View the online version of the campaign
+ Follow them on Twitter

Want to see a few more examples? Take a look at the slideshow below.


Things we love: Themed email newsletters

See how Choke Design Company introduces their new monthly mailing

Take our survey for a chance to win a custom design or free emails

Tell us about your email marketing goals, challenges and more so we can help with your holiday plans
Emma Holiday Survey
One lucky survey-taker will win. Click above to take the survey.

My Nashville coworkers remark on the strangeness of planning for the holidays when it's still 80 degrees outside, but here in Portland, the weather has taken a decided turn toward autumn. With a chill in the air, I can't help but think of scarves and boots, pumpkin-flavored lattes and roasted sweet potatoes. And, yes, even holiday-themed emails. The holidays start early at Emma, as our design team gets excited about crafting holiday designs for our customers (from fall-inspired stationery to more traditional Christmas postcards), and as our team of bloggers plans a slew of fresh email tips and resources. (Last year's included, among other things, holiday subject lines and a video of holiday email ideas.)

This year, we'd love your help as we get into the holiday spirit. We want to know what you want to learn this season and how we can help you reach your email marketing goals.

Take our 10-question survey here.

Want another reason to feel merry? By submitting the survey, you're eligible to win one of three prizes of your choice: a custom holiday design, custom signup form for your website or a free month of emails. We'll announce the winner in November, along with results of the survey. Now, go get yourself a pumpkin latte.

Turning clients on to opt-in email marketing

For Emma agencies, educating clients on the importance of permission standards and list hygiene is critical. But that doesn't mean it has to be stressful.

Send great emails to people who want to receive them. That's the phrase that rings in my head when I ponder how an agency speaking with a prospect might summarize the goal of permission-based email marketing. Pretty simple, right? The "great emails" part is easy to communicate, especially if they've seen samples of your work, or worked with you on other ventures. So now we come to the "people" part. Who are these people? Where did they come from?

This is where the conversation with a prospective client will turn to permission, and it's one of the most important conversations you'll have. Fortunately, our permission policy is pretty straightforward: To receive your emails, people on your list…


+ Someone who is a customer, member or subscriber of your business or organization

+ Someone who has specifically asked to receive your emails by opting in or signing up in some way or

+ Someone who has bought a product or service from you in the past 18 months.

Emma's permissions policy is in place to minimize the risk to our customers' sending reputation, and to maximize the effectiveness of your email strategy. But the reasons for these standards may not be immediately apparent to a client who's considering renting or purchasing a list, or attempting to send to an audience full of email addresses that they haven't contacted in years. So when you're getting to know a new client's email list, it's important to communicate the value of a healthy, permission-based list. Here are a few points to focus on:

A healthy audience means a healthy sending reputation: In the United States, anti-spam legislation doesn't go as far toward limiting unsolicited messages as most email service providers (like us) do. It's not illegal to send unsolicited emails as long as you label them as ads, include your business address and offer a way to opt out of your list. But that doesn't mean it's a good idea: Email traffic is privately monitored, policed and regulated well beyond the limits of CAN-SPAM regulations. Most email clients and servers have their own rigorous standards to protect their users' inboxes from spam overload. And while your client may not realize it, their business builds upon their reputation with servers and spam blacklists with every mailing. Email campaigns to well-maintained, opt-in lists tend to yield high delivery, open and click-through rates, all of which look like little gold stars next to a company's name. On the other hand, mailings to rented, purchased or outdated lists are pretty easy to spot. They consistently yield low delivery and open rates, and high rates of opt-outs and spam complaints. Servers and spam filters take notice — a company with a history of poor mailings will have a notably harder time reaching their audience's inbox. In fact, they often have a hard time reclaiming their good name even after they've seen the error of their ways and changed their practices.

Our own reputation as the "carrier" of your clients' messages also comes into play. To maintain our exceptional delivery rates, we work vigilantly to ensure that all the emails we send are in line with our policy, so sickly-looking response rates are often a sign that we need to step in to diagnose the cause. In situations where we need to find out more about an email delivery issue, we always start by learning more about the email list. In most cases, the source of the problem lies with the manner in which the email addresses were gathered.

Smart list practices yield a higher return on investment: In the early days, a good deal of conversation around email marketing revolved around accumulating the biggest email address list possible. Growing your audience is still an essential piece of any smart marketer's strategy, but in recent years, email marketing specialists have increased their focus on engagement over simple accumulation. A good subscriber's initial points of engagement came before they saw their first email — they bought a product or service, or they asked to hear from you. There's a connection to build upon. An audience that hasn't made this connection, or made it so long ago that they've likely forgotten about it, won't be as responsive to your message.

A good email list is naturally grown: There aren't any shortcuts when it comes to building a healthy, responsive email list. But there are plenty of great ways to attract the right people to join your audience. Find the places where your intended audience makes contact with you, and make sure there's an opportunity to sign up waiting for them there. For some businesses, this may be a fishbowl for business cards at the cash register. For others, it may be on your company's homepage. If your client is a heavy emailer, have they considered including a signup link in their signature? As you grow your audience, find ways to enable them to help you find new subscribers. Tweet links to your email campaigns and post them on Facebook. Encourage subscribers to share your emails with friends. Give them options to choose to receive the content that they're most interested in. There's nothing like watching your email audience grow healthy and strong. If your client can get excited about that prospect, they're already well on their way to finding the people who want to hear from them the most.