Skip to content
Bot Blog

Mar 09, 2021

How To Use Multivariate Testing To Supercharge Your Email Performance

Email performance can be tricky to measure. Here are tips on how to A/B test for your very best email marketing results.

If you use email marketing, you’ve probably spent longer than you care to admit debating whether or not to include an emoji in your subject line or whether the ‘Order Now’ button should be red or green. You know these small factors can make a big difference in the open, click-through and purchase rates of your emails… if only there were a way to look into the future and predict which tweaks your audience would respond to best. 

Enter: multivariate testing. 

Multivariate testing is an email marketing feature that allows you to predict which of a series of changes will lead to the best results and tailor your campaigns accordingly. Here, we’ll explain why multivariate testing is so beneficial in email marketing and how you can use it to drive more engagements and greater revenue from every email you send.

What is Multivariate Testing?

You’ve probably heard of A/B testing, a conversion rate optimization tactic in which two similar versions of a marketing item like a landing page or email are tested against one another. A/B testing generally refers to just two versions of the item being tested. In multivariate testing, you can test many different variations on the same content piece. 

The great thing about multivariate testing for email is that you establish the results quickly, before your entire audience receives the campaign in question. This way, you can ensure as many subscribers as possible are sent the “winning” campaign. 

Here’s how it works. First, you set up your campaigns. Each of them will have slightly different variations to things like the subject line or pre-header text. Once your variables have been established, you’ll set a test duration and a sample size. This is the “test” population. 

During the test duration, i.e. one hour, your sample audience will receive the different variations of your email in equal proportions. So, if you had a sample size of 600 and three options to test, each of the options would be sent to 200 subscribers.

After the predetermined time frame is up, the results are analyzed to determine which version of the email is the winner. You can base the “winner” on a number of metrics, including open rate, number of clicks, or resulting revenue, depending on what’s most important for that campaign. Once you’ve established a winner, that email will be sent to the remainder of your audience, maximizing the campaign’s likelihood of success. 

What Should I Test With Multivariate Testing?

Image by Pete Linforth from Pixabay

Subject line and preview text. Test out different lengths, topics, word choices, use of emojis, and use of the customer’s name. 

‘From’ name. Try different variations of the company’s name, like ‘Acme’ versus ‘Acme Furniture Company,’ or use an individual’s name, like ‘John at Acme Furniture.’ 

Email design. Long or short? One column or two? Image heavy or text only? Test header sizes, font sizes, logos, image positions, and image variations. 

Colors. This might include the color of your headers, accents, background or calls to action.

Call to action. Try changing the phrasing, format (button versus text), size, and CTA placement within the email. 

4 Best Practices For Boosting Email Performance With Multivariate Testing

Use an Established List

In order for multivariate testing to be effective, we recommend using it on lists of 5,000 subscribers or more. This ensures you not only have a large enough pool from which to draw an adequate sample, but that your results aren’t skewed by your list being too new. If the majority of people you’re emailing aren’t used to hearing from you on a regular basis, this could result in higher-than-normal unsubscribe rates. 

Have a list that’s smaller than 5,000 subscribers? No worries! You can still take advantage of conversion rate optimization. In this case, we’d recommend doing a simple A/B test where half of your subscribers are sent one variation of a message and half are sent another. The results can be analyzed and used to make decisions in future campaigns. 

Use a Reliable Sample

This is another reason the list you start with needs to be of adequate size. To get results that are considered statistically significant, we recommend a sample audience of at least 500 test subjects. 

Within any sample for any type of test, there’s going to be random variation (i.e. results that are based purely on chance rather than being indicative of a meaningful trend). The smaller the size of your sample, the less reliable it is. For most lists, a sample audience of 10 to 20% is a good benchmark. 

Test One Variable At A Time

While multivariate testing allows you to test as many variables as you want in any given test, your best best is to test just one item at a time (i.e. subject line or email design, but not both). The reason being that if you test multiple variables, there’s no clear way to know which of them contributed to the winning results, which is important intel if you want to continuously improve your email performance over time. 

Additionally, the more options you test, the bigger sample size you’ll need, as each additional option splits the target audience into smaller test groups. 

Examine Results Over Time

Image by Lucas Wendt from Pixabay

Ideally, multivariate testing should be just one tactic in your overall email optimization strategy. There are numerous factors that can influence the results of a multivariate test, from the day of the week to the time of day of your send, holidays and even external factors like major weather events. Holiday season in particular is a tricky time to make broad assessments based on multivariate tests because subscribers’ email behavior is different than normal. 

So, don’t just run one test and think of it as gospel when it comes to future subject lines/email layouts/etc. Instead, conduct tests on an ongoing basis and use the results to expand your ever-growing pool of knowledge about what resonates best with your subscribers.

Supercharge Your Email Performance with Multivariate Testing from Springbot

Springbot is pleased to announce the rollout of multivariate testing, the latest tool in our user-friendly tool kit to help marketers get the most out of every campaign. Just a few clicks is all it takes to set up an audience sample and determine your testing variables. 
Learn more about how to set up your first multivariate test here. New to Springbot? Contact us today to schedule your free demo and learn how to harness the power of automation and personalization for more profitable marketing on every channel.

Latest Articles

17 Alternative Ways to Grow Your Email List Outside of Your Website
Email Marketing Strategy

17 Alternative Ways to Grow Your Email List Outside of Your Website

Discover 17 alternative ways to build email lists without a website. Yes, it's possible! Learn how with Springbot, your email marketing all...

July 26, 2023

Explore 40 Different Email List Building Strategies for Organic Growth
Email Marketing Strategy

Explore 40 Different Email List Building Strategies for Organic Growth

Unlock the Power of Organic Email List Building! 40 Proven Tactics for Engaging Content, Social Media Strategies & Effective Referral Progr...

July 26, 2023

Rekindle Lost Customer Relationships with Winback Email Subject Lines
best practice for email subject lines

Rekindle Lost Customer Relationships with Winback Email Subject Lines

Rekindle lost customer relationships with winning Winback Subject Lines. Explore 100 different email subject lines to craft the perfect win...

July 26, 2023