1. Introduction

An optimised email programme will generate more revenue and build greater engagement. It will add to the company’s understanding of its customers, making the channel an even more valuable resource, worthy of greater investment.

Testing is the most common method of optimisation. Many email service providers now include basic testing modules in their platforms, which makes testing more affordable and accessible.

These gains are more likely to be realised if the email team follows a regular testing programme based on established scientific principles and processes in order to create meaningful tests. This manner of structured testing is more likely to generate meaningful, statistically significant results.

This guide outlines the importance of email testing and covers different testing approaches. It describes the steps involved in a holistic testing programme that takes a comprehensive view of the whole process and follows scientific protocols, from developing a hypothesis to interpreting the results.

1.1 Executive summary

This guide sets out frameworks and best practice for email testing and optimisation. Key learnings include:

• Email testing can reveal deep insights about the company’s customers and what motivates them to act on emails and stay engaged with the brand.
• Results and findings from email testing can have applications beyond the channel. The insights gathered can be used to inform testing plans for other channels. Testing can also show whether the behaviour and motivations that drive email customers also apply to the customers in these other channels.
• A formal testing structure based on scientific processes and protocols will deliver more reliable results. All testing should begin with a hypothesis and use appropriate sample sizes to ensure results are statistically significant.
• Holistic testing is a method that looks at the entire testing process as a whole, instead of each working part individually. It focuses closely on what motivates customers to act as they do and seeks both short-term and long-term gains.
• Marketers who wish to avoid appearing intrusive in triggered campaigns such as browse abandonment may consider testing whether covert personalisation, which avoids overt references like the customer’s name or browsing details, generates more positive responses.

1.2 Methodology

The methodology involved two main phases:

• Phase 1: Desk research to identify relevant issues, examples and models.
• Phase 2: A series of in-depth interviews with a range of email marketing experts.

Econsultancy would like to thank the following interviewees who contributed to this report:

• Tony Allen, Global CRM Lead, Volvo Cars
• Jess Evans-Tudor, Head of CRM, Rated People
• Kelly Haggard, VP – Marketing Innovation, Synchrony
• Jakob Holm, Email Marketing Director, Pandora
• Mar Juárez, CRM Manager, IVI RMA Global
• Beata Linz, Global CRM Director, AdBirds
• Lisa Pettit, CRM Lead Developer, Specsavers
• George Williams, CRM Manager, FARFETCH Platform Solutions

1.3 About the author

Kath lives and breathes email marketing and is recognised as one of the industry’s leading email marketers. As CEO of boutique consultancy, Holistic Email Marketing, she devotes her time to developing customer-centric ecommerce journeys using a holistic, multi-channel approach.

Winner of the 2021 ANA/EEC Email Marketers Thought Leader of the Year, Kath is an industry veteran of 23 years. She is the author of the bestselling and award-winning book Holistic Email Marketing, and is a sought-after international keynote speaker, blogger, expert contributor and trainer.

Linkedin.com/in/kathpay

kath@holisticemail.com

@kathpay

2. The Importance of Email Testing

The ability to optimise easily is one of email’s greatest advantages over other marketing channels. Marketers that routinely send messages to their subscription lists have regular opportunities to test and optimise. This allows for continuous improvement.

Testing can be conducted on all types of messages, including promotional campaigns, newsletters or updates to whole subscriber lists, targeted messages that match subscriber interests and preferences, and personalised messages triggered by subscriber or customer behaviour.

Email testing and optimisation can form part of the quality-assurance workflow before a campaign goes out, or it can be part of a dedicated programme of experimentation that can deliver greater insights and drive improved performance. This guide will focus on the latter – on ways marketers can begin or advance their testing programmes to optimise all aspects of their emails.

An optimised email programme will generate more revenue (thus offsetting the costs of testing and analysis), build engagement, add to the company’s understanding of its customers, and make the email channel an even more valuable resource, worthy of greater investment and respect.

2.1 What email testing can reveal

Every message sent is an opportunity to learn more about the audience, their likes and dislikes, and what motivates them to act. Every insight learned from a test can help improve the entire email marketing programme.

Much advice about email testing focuses on immediate results: which subject line drove more email opens? Which button colour do people prefer? Did people click more on image A or image B? However, email testing can go much further.

In addition to unearthing deep insights about the company’s customers, testing can reveal whether the marketing team is investing in the right email strategies and tactics, or if it is wasting time and money on things that do not work.

The latter is usually a direct result of what happens when marketers fail to let strategy lead the way and instead focus on tactics without a plan, and without testing them before implementation.

A well-designed and executed strategic plan will be based on the insights gained from regular testing and optimisation. For this reason, marketers are better served when they take time to test and analyse for long-term insights. They might not be as immediately rewarding as learning that customers click or tap on blue buttons more often than red ones, but the long-term results will be more effective.

2.2 Applying insights beyond the channel

Email marketers have a unique advantage over their website or social media marketing colleagues. Thanks to the fact that email is by nature a ‘push’ channel, email marketers do not have to wait for customers or prospects to find them through organic or paid search, or just by typing a URL into a browser.

Email subscribers are a built-in target market. They have expressed at least some interest in the brand. The permission-based nature of email opt-ins means that subscribers choose to receive a company’s emails. This makes them the ideal audience for testing ideas.

Marketers can the use the results from these tests as a starting point for testing across the company’s other marketing channels. They should not be used as a blueprint or substitute for channel-specific insights, as people in those other channels are likely to be at different stages in the buying cycle. Rather, the results and information gleaned from email testing can help guide testing plans for these other channels and be used as an additional resource for marketers. Testing can also show whether the behaviour and motivations that drive email customers also apply across these channels.

2.3 Marketing leaders prioritise experimentation

In the 2021 Marketer Email Tracker Survey published by the Data & Marketing Association, only 48% of marketers rated their organisation’s testing as ‘Leading’ or ‘Good’, and 14% said they were doing no testing (Figure 1).^[1]

Figure 1: 14% of marketers do no testing

Source: DMA Marketer Email Tracker 2021^[2]

The ability to build and test experiments is one of the telling factors that separate marketing leaders from the rest, according to findings in Econsultancy’s 2022 Digital Trends: Experience Index, published with Adobe.

The study found that 61% of marketing organisation leaders rate their ability to build experiments to test in the field as ‘strong/very strong’, in contrast with just 21% of mainstream and 5% of laggard organisations.

“Our study uncovered a yawning chasm between leading marketing organisations and the rest of the pack,” David Carrel, Vice President, Marketing, Adobe Experience Cloud, wrote in the study. “Mainstream and laggard organisations must double down to close the gap before the leaders pull even further ahead.”

3. A Framework for Email Testing

Email testing is both art and science. Creative thinking helps drive the possibilities for testing, but science ensures that the results will be reliable and repeatable. It relies on a formal structure that identifies a problem, develops a framework to address it, follows a strict procedure that eliminates chance, error or bias, and delivers results that can be repeated.

Marketers who do not test or who test only easy, peripheral things like subject lines or calls to action are missing out on helpful information that could guide fresh efforts to improve everything from an individual email campaign to an entire email programme.

Advanced testing skills are important to learn. Whether marketers are establishing a testing programme or upgrading or optimising a current programme, following the steps outlined below will help them create a testing plan that adheres to scientific standards, aligns with testing objectives and delivers reliable results.

3.1 Step 1: Identify the objective

In email testing, as with testing in any other scenario, it is essential to understand first what the objective for the testing will be. This will guide everything from the hypothesis, which states what the outcome of an experiment might be, to the testing protocols and the measurements used to success or failure. (See the Econsultancy’s report on Email Marketing Best Practice: Setting Goals and Measuring Performance for more on this subject.)

If the objective is to both gain an uplift on the immediate campaign and gain long-term insights, the objective of the test will be to gain audience insights as well as business insights. What motivates the audience to purchase – loss aversion or benefits? What will drive up average order values? How can the brand get more traffic for less money? (For further guidance on the psychological principles that drive people to act, see Econsultancy’s report on Email Marketing Best Practice: Copywriting, which covers techniques of persuasion.)

3.2 Step 2: Set up the testing plan

This step has four parts:

• Create a hypothesis. Successful testing takes more than an educated guess and the combination of some random variables. Formulating a hypothesis provides a more structured basis for the testing plan. The hypothesis says: “I believe making X change will cause Y to happen because of Z.”
  
  .
  
  Each part of the hypothesis is important, but it is essential to include the ‘because’ to understand why the campaign delivered the results it did.
  
  .
  
  The hypothesis is the statement about what might happen when a number of testing factors come together. This is the basis of the test because it identifies what will be tested, how results will be measured, what technology or channel will be used to perform the test and what the expected outcome will be.
  
  .
  
  Here is a sample hypothesis: “Loss aversion copy is a stronger motivator towards conversions than benefit-led copy because multiple studies have proven that people hate losing out more than they enjoy benefiting.” (For more hypothesis examples, see Section 5.1)

• Select the metrics to measure results. These metrics should map back to the hypothesis and business objectives. Open and click rates are easy to measure, but they might not be the metrics that will help to determine whether results prove or disprove the hypothesis.
  
  .
  
  Depending again on the campaign objective, the appropriate metrics might be conversions, sales, average basket value and the like. (See Econsultancy’s report on Email Marketing Best Practice: Setting Goals and Measuring Performance for a detailed discussion of metrics.)

• Select the factors to test. Establishing a hypothesis will help to choose the factors to test, such as subject line, message copy, offer, call to action or frequency. It might also help in selecting multiple factors, which can lead to more long-term insights instead of the ‘moment in time’ results when testing on a single factor.
• Ensure results are valid. Test results must be statistically significant and not simply a random occurrence. It is necessary to choose a large enough sample size from a customer database to ensure the results are valid. This can be achieved by using a split test calculator, a digital device that can help you identify how many email addresses would need to be included to reach a statistically significant finding. Statistical significance means a 90% to 95% chance that test results reflect the conditions of the experiment and not because of random occurrence or audience selection bias.

3.3 Step 3: Analyse the results and apply the findings

Record the results and findings in the testing plan. The results will determine whether it has been possible to prove or disprove any hypothesis, while the findings are the insights that are drawn from the results. For example, the results could be the open rate percentages for two test variations, while the finding would be that a particular variation triggered more opens.

Depending upon the hypothesis and the factors tested, some findings will apply not just to the test run for the emails themselves but also across all digital channels. That is one of the great benefits of email testing. An email audience is ideal for sampling because it is readily available in the customer database. Email provides a solid basic testing structure that can be built on to sharpen insights and improve marketing efforts bit by bit across all channels.

Having analysed the findings, there will be the potential to use what has been learned in web copy, search keywords, PPC campaigns and landing pages, and in banner ads and copy that are run on third-party sites for retargeting or network campaigns.

4. Approaches to Email Testing

Advances in technology, particularly at the ESP level, should mean that testing becomes more widespread and with less drain on resources. The creation, launch and adoption of AI-based testing tools has also been a major factor in giving marketers more control over the testing process, including allowing direct access to testing and results.

Different attributes of an email campaign or newsletter should be tested in a systematic way to determine factors that improve campaign responses. The underlying purpose is to improve results in a second wave or for future campaigns.

4.1 Testing formats

The two main testing formats marketers can use are as follows:

• A/B split-testing – An email audience is divided in half at random. The control version of the email – usually the default or primary version – is sent to half the audience, and the test version is sent to the other half. The marketer either watches the results and uses them as learnings in future campaigns or stops the campaign send once a clear winner emerges, and then resumes by sending the winning version to the remainder of the audience.
• Multivariate testing – As in an A/B test, the email audience is spilt and shown different versions of a design, only this time multiple elements are tested simultaneously. This gives insight into how the variables interact with one another. It is a useful method for testing template designs or combinations of text and calls to action.

In addition to these formats, marketers can choose to conduct a pre-send test, where a subset of the list is used to determine the best combination of offer and creative to use in an email before sending it out to the rest of the list.

4.2 Pre-send testing

Pre-send testing is the practice of running a trial of a campaign in which different elements are varied with a subset of the audience, before rolling out the campaign to a wider audience once the best combination of campaign variables has been determined. For example, emailing two audiences that each represent 10% of the total list to find which offer works best.

It is important to a) have a large enough sample size to test with, and b) decide which metric will be used to determine the winner.

With pre-send tests, the metric (opens, clicks or conversions) that will be used to decide the winning version should be made clear. Taking a subject line split test as an example, the version that generates the most opens might not always result in the most clickthroughs or transactions.

When pre-send testing, it is necessary to decide how long to wait before picking which version to run with, and when to deploy the remainder of the campaign.

Pre-send tests are possible with most good ESPs and allow marketers to see real open and clickthrough rates from a smaller sample volume before deploying the whole campaign volume.

Table 1 shows what results from a pre-send test might look like.

Table 1: Example of results from a pre-test

Nine steps to set up a pre-test

1. Create a hypothesis, which will guide every step of the testing plan. Review the steps in Section 3.
2. Create a testing plan, which is a document that lists the hypothesis, the email objective, success metrics, factors and audience to be tested, the variables and controls, metrics and conclusions. Set up a spreadsheet with these fields or download the planning template developed by Holistic Email Marketing.^[3]
3. Split the list into at least two parts, also called cells, including test and control. The control cell is similar to previous communications.
4. Ensure each test cell supports one of the hypothesis’ testing factors, such as ‘benefit’ or ‘loss aversion’. It is crucial that only one feature or variable is changed for each test cell.
5. Ensure the test group is smaller than the size of the list but sufficiently large for differences in response to be significant.
6. Take the test group randomly from the list. The list should contain a range of segments and lengths of email subscription. Taking the first ‘X’ amount in a list will often give those who have been on the list the longest.
7. Test responses by adding identifying links in the creative copy to indicate which link a respondent clicked. Alternatively, separate landing pages could be created, which is more straightforward.
8. Use the best combination of variables according to response when emailing the remainder of the list.
9. In order for results to be proven valid, it is necessary to allow at least half a day, or the length of time required to achieve statistical significance, before making a decision. A couple of hours is never enough. Pre-testing when using conversion rate as the success metric will generally prove difficult due to the time it takes for conversions to happen.

Following the pre-test, review all metrics returned in the test, not just the metric selected to determine success. For example, with a subject line test, most marketers would normally use open rate as the success metric. However, if the subject line has a strong synergy with the main part of the message, it might get a higher clickthrough rate than an email with a different subject line but higher open rate.

Another example would be offers that generate a high clickthrough rate but do not convert very well when the recipients see more detail on the landing pages.

A note on using open rate as a measure

As discussed in Section 3.4.2 of Econsultancy’s guide to Email Marketing Best Practice: Setting Goals and Measuring Performance, open rate is not always the best measure of the success of a subject line. Some can drive high opens but few conversions. The most appealing subject line might draw a large number of opens from customers who, upon reading the email, find they are not interested in the offer. Or, they click to the landing page, only to drop out there.

The example below is from a test run for a company.

The hypothesis said: “A subject line that promotes all the savings to be had with Brand X will deliver more conversions than one stating the broad benefits because our customers are very focused on savings.”

The control email has this subject line: “See how Brand X will save you money!” The variant email’s subject line is “See how Brand X saves you time with worry-free shopping!”

The results in Table 2 showed that the “worry-free shopping” benefits won with a 0.6 percentage point uplift in conversions, with 99% statistical confidence, even though the open rate was similar between the two versions and had no statistically significant difference.

Table 2: Testing subject lines – the results

Source: Holistic Email Marketing

There should be a great deal of importance placed on the ‘success’ metric because that will lead to measuring the right things in any test. A subject line that generates a low open rate might actually drive more conversions. This is why it is so important to have multiple criteria for measuring the success of a test.

When testing, each test can be marked with key codes to make it clearer which test condition resulted in what response when looking at the web analytics tool.

For example, these codes can be used to highlight the characteristics of each test:

• List = L1
• Segment targeted = S1
• Date/time broadcast = D1
• Offer = O1
• Subject line = SL1
• Content = C1
• Hyperlink = H1 (e.g. first hyperlink in message)
• Format = FT, FH
• Call to action = CA1

These codes can then be combined with the hyperlink that is the call to action within the email. A question mark is used to separate out the testing key codes from the landing page to understand the clickthrough..

4.3 Live A/B split testing and multivariate testing

Live A/B split testing is easier to conduct than pre-send testing because the list is divided equally among different test cells during broadcast to the whole list. In this approach, the success of different creative treatments is tested during the campaign in order to learn for future campaigns. This is one of the best testing methods since it is relatively fast and cheap.

A common approach is to do a simple two-way split test (50%/50%), in which half of the list receives either the control or variant A, while the other receives variant B.

Multivariate testing is best known for testing landing pages, for example where visitors are referred from paid search. However, multivariate testing can also be applied to test different creative elements of an email. It is only generally worthwhile and statistically significant with template redesigns for a large list. Individual elements are tested through substitution into a template.

4.4 Key considerations for testing

• Consider a pre-send test. When a large volume of subscribers is directed to a single landing page, it may be worthwhile performing a pre-send test.
• Statistical confidence level. It is necessary to check the confidence level is statistically significant. When testing, be aware that it is necessary for a large enough sample and large enough response to make any results statistically significant. The aim should be for a 90 to 95% confidence level of statistical significance.
  
  .
  
  A discussion of statistical significance is beyond the scope of this report. Readers might find Mailchimp’s guide, A Business Owner’s Guide to Understanding Statistical Significance,^[4] useful. It is an updated and marketer-focused resource that explains statistical significance in detail and offers guidance on avoiding errors.

• Random sample. Ensure the test sample is random. Do not include older addresses or certain segments in one cell and different types in another. Each cell must be selected on random criteria.
• Consider real-time optimisation. For high-volume broadcasters, subject lines or creative that delivers the best response can be automatically optimised during the campaign.
• Ask users directly. With much statistical analysis, one area that often gets overlooked is feedback from real people, so consider asking users directly for their opinions on email communications. Do not just ask about the email layout or copy. Ask how the emails support the brand. A research programme could survey email subscribers on these factors:
  • Email relevance
  • Email believability
  • Likeability (content)
  • Likeability (design)
  • Future intentions (multichannel)
  • Comparison with competitors
• The answers to these types of questions can be received in focus groups, or through online survey tools and eye tracking studies. Often the subject line ‘Tell us what you think’ will gain the highest response rates. For further advice on undertaking research please see Econsultancy’s Fundamentals of Market Research and Insights Best Practice Guide.
• Use generative AI to create variations for tests. GenAI tools, such as ChatGPT, Midjourney and Jasper, can be used to create copy and image variants for testing. For example, the tool could be asked to produce variations of copy that reflect different emotions or psychological motivations (loss aversion, reciprocity, etc.) or a hard-sell direct approach. For advice on writing prompts and more information on using generative AI tools to create variations for testing, see Econsultancy’s report on Email Marketing Best Practice: Copywriting.)

4.5 Create a framework for continuous testing and optimisation

Successful testing is not a ‘one-and-done’ proposition. After a team becomes proficient in running a testing programme, it is recommended to create two permanent streams in the automated programme, Stream A and Stream B.

Every time a stream becomes a winner based on a hypothesis, a new test should be created with a new hypothesis. The individual email campaigns will be updated accordingly and run with the new test. This creates continual optimisation, and the learnings can be utilised wherever they are applicable.

Testing is an essential aspect of email marketing because it can be used to measure whether marketing teams are achieving what they set out to do and whether they are using their email resources wisely. It helps marketers apply rigour and discipline to their email programmes and shows what is working, what is not, and where the path to success lies.

5. Setting Up an Effective Email Testing Programme

Holistic testing is a concept developed by Holistic Email Marketing. It can be implemented as a framework that imparts both a testing structure and a method for applying the results generated from those tests.

Holistic testing takes a comprehensive view of the entire testing process instead of focusing on individual ‘working parts’. It looks closely at what motivates customers to act as they do. It is not enough to know that Subject Line A was associated with more conversions than Subject Line B – marketers need to know why the email with that subject line was more appealing to customers.

Holistic testing also incorporates more variables, such as testing a subject line, a headline, and the call to action, as long as they all support the hypothesis that forms the basis for the test. This is because the goal is not to see which subject line prevailed, but which combination drove the results that the hypothesis predicted. This makes the results more robust and gives them more widespread application, as well as a longer shelf life than an ‘ad hoc’ or ‘one-off’ test would.

As with any robust testing programme, holistic testing uses a scientific methodology to ensure the results are reliable. It also adopts a mindset in which every email is a potential survey of a brand’s audience.

As the email database is a proxy of the brand’s target market, customer actions on the brand’s emails are their votes on what resonates best with them. These actions can include any or all of the following:

• Opening the email
• Clicking on the email’s call to action
• Clicking on the unsubscribe link
• Clicking on other links in the email
• Using a forwarding link in the email to send the message to others
• Opening the message but not clicking
• Deleting the message without opening it

Hence, many of the insights revealed via testing in email will also apply to other marketing or communication channels.

Two key benefits of holistic testing are as follows:

• It prioritises long-term gains over short-term wins. Holistic testing uses regular, systematic testing and builds upon previous insights. It delivers not only immediate short-term uplifts but also valuable insights into the organisation’s audience, helping marketers to understand what works best for the email programme and, indeed, the overall marketing programme. These long-term gains deliver a solid foundation for consistent performance and incremental innovation – the aggregation of marginal gains that can lead to lasting improvement.
  
  .
  
  Holistic testing does not rely on top-of-funnel metrics, such as open rates, to measure success. Instead, it factors in many metrics and conditions into its insights, providing more reliable information to use when measuring the overall success of an email programme.

• Testing across multiple campaigns gives marketers a larger and more reliable sample size. Relying on campaign-level metrics means learnings are restricted to that campaign only. Using the same hypothesis across multiple campaigns gives a larger sample size. Thus, the results can give additional context to a metric that applies to the lower end of the marketing funnel, such as conversions, instead of being limited to a top-of-the-funnel metric like open or clickthrough rate.

5.1 Ten-step holistic email testing plan

Holistic testing is not just about the test result. It is about understanding how to use the test result in combination with others to learn more about the audience – who they are, what motivates them to act, what does not motivate them. Having this information quantified via regular, rigorous testing can lead to deeper insight into an audience, which can then be applied to other marketing channels to guide decisions all through the company.

This plan builds on the scientific testing framework outlined in Section 3.

1. Identify the objective

Any test, even the most basic A/B subject-line test, performs better when marketers have identified what is to be learned from it. The underlying objective will guide all the decisions to be made, from choosing the variables to how long to run the test and how to measure and apply the results.

The objective can have both short-term and long-term impacts. A short-term gain would be to learn which email combination generates more conversions, while a long-term gain would be to identify what motivates or resonates with the audience. This long-term finding can then be put to use across other channels.

2. Identify the question to answer or the problem to solve

Testing for the sake of testing will only waste time and money. Identify the question or problem by finishing this sentence: “I wonder why…”, or by addressing a challenge such as “How can we persuade more people to move from the free to the paid version of our product?”

3. Formulate a hypothesis

The hypothesis takes the question being asked and formulates it as a theory: “We think our customers are motivated more by avoiding loss than by gaining benefits through acting on our emails.”

The hypothesis guides the decisions to be made in the following steps. Figure 2 below lists three sample hypotheses, along with elements to be tested and where results can be applied beyond email.

Figure 2: Three sample hypotheses and applications

Source: Holistic Email Marketing

4. Select the factors to test

The hypothesis will dictate which factors are most appropriate for the test, but these could include copy elements such as the subject line, call to action and message; design elements, such as the images and layout; or tactical elements, such as personalisation, timing and frequency.

5. Identify the success metrics

These metrics must align with the hypothesis and objective. While open and clickthrough rate are easy to measure, business metrics that are tied to conversions, sales or average basket value, for example, may be more appropriate.

6. Let the test run long enough to achieve statistical significance

Cutting off a test too early can render any results useless. A test should be allowed to run until it reaches a large enough sample size from the customer database or subscription list, especially when using conversion rate as the success metric. For example, if the bulk of conversions happen on days two and three after a campaign is deployed, it is pointless to perform a pre-send test which selects the winner after three hours.

Use a sample-size calculator to determine how many results are needed before results from the test can be reliably measured. Many online sources provide free tools for this purpose, such as Zettasphere’s calculator for A/B split testing.^[5]

7. Repeat the test at least three times

To ensure a test result is not an anomaly, it should be run several times to determine that the winning result is not a fluke. This is not feasible under ad hoc testing as it would mean repeating the same subject line and copy for multiple campaigns. Instead, the same test can be run multiple times using different copy as long as it supports the hypothesis. The concept is being tested here (i.e. benefits versus loss aversion), not the specific wording.

8. Record the test framework and results

As soon the test is complete, the details and results should be recorded in a testing planner. These are some of the elements to include:

• Total emails sent
• Open rate
• Click-to-open rate
• Products purchased
• Conversion rate
• Average order value
• Statistical confidence
• Conclusion
• Recommendation

9. Analyse and learn

The testing team should study what the test revealed and what has been learned from it. Were the results expected or were they inconclusive or contrary to expectations? If the test setup and operations were satisfactory, the team members who worked on it can record their findings. The record should also indicate whether the results support the hypothesis and what recommendations can be drawn from them. The team should also consider how the findings might apply to future campaigns as well as other areas of marketing, so that the results are not just looked at in isolation.

10. Roll results out to other channels to use as a starting point for testing

This will be possible if the test has been set up correctly and allowed to run long enough to achieve statistical significance. At this point the learnings from the test can be applied to marketing efforts in other channels as a starting point for further testing, or as reference material to be considered when planning other marketing programmes or campaigns.

5.2 Testing in automated email programmes

The previous guidance has been given mainly for broadcast promotional campaigns (one to everyone) or targeted/segmented campaigns (one to everyone in a segment). Holistic testing also can be used for  automated programmes.

The procedures are similar, but they have one key difference: automated messages are sent out in response to a trigger, such as behaviour, and not in a batch, as with an email campaign. Examples include a welcome/onboarding email or series, an abandoned-cart or browse reminder, a webinar registration or white-paper download, and the like. The testing process must wait until enough of this kind of message has been sent to achieve a statistically significant sample size.

However, marketers must resist the temptation to end it as soon as it reaches statistical significance. This test needs to run until each variant has achieved the recommended sample size as directed by the sample-size calculator used to determine how many triggered emails must go out.

6. Conclusion

Creating a culture of testing is essential if email marketers want to continue to deliver measurable returns. Relying on gut feelings or the highest paid person’s opinion (the HiPPO effect) can lead to disappointing results. This is because even the smartest and most experienced people will frequently be wrong about what an audience will prefer. For that reason, as many disputes as possible should be settled through testing programmes, such as A/B testing or multi-variate testing.

Testing and optimisation is necessary for understanding whether a marketing initiative is repaying the time and money invested in it, whether in actual monetary terms or in efficiencies, added insights, or a better customer experience.

Many brands do not test enough, or only focus on the same handful of email elements, such as subject lines and calls to action. Marketers have a wide range of tests to call on, and as marketing has become more data-driven, more brands are likely to have access to data for testing and an ever-increasing array of tools for testing quickly and accurately.

Testing and optimisation might add time to the email workflow for a campaign. Executives who fear the potential loss of revenue from audiences that do not see the ‘winning’ version of an email campaign must be persuaded that testing pays off in the long run.

6.1 Key takeaways

• Email testing can reveal both short-term wins and losses along with long-term insights that can support or revise email strategy and guide the use of tactics.
• Repeated tests deliver more accurate and useful insights than ‘ad hoc’ or ‘one-off’ tests where results might not be repeatable in subsequent campaigns.
• Use a hypothesis for all tests and be sure to include a ‘because’, since this will be the insight about an audience that can then be used across other channels.
• Build in a testing plan for all campaigns and automated programmes.
• Record the results and be prepared to test multiple times to ensure the result is not an anomaly.
• Ensure the result is statistically significant. If testing a campaign, the sample size tested will be key. If it is an automated programme, the length of time the programme is run will be key.
• Be sure the metrics used to measure the outcome of a test link back to the hypothesis being tested.
• Email testing works best when it is used to reveal long-term insights.
• Holistic testing takes a comprehensive view of the entire testing process, not just testing components, and focuses on what motivates customers to act as they do.

7. Further Reading

7.1 Econsultancy email marketing reports

Econsultancy | Quick Guide to Why Email Still Matters

https://econsultancy.com/reports/quick-guide-why-email-matters/

Econsultancy | Email Marketing Best Practice: Setting Goals and Measuring Performance

https://econsultancy.com/reports/email-marketing-setting-goals-measuring-performance/

Econsultancy | Email Marketing Best Practice: Segmentation and Personalisation

https://econsultancy.com/reports/email-marketing-segmentation-personalisation/

Econsultancy | Email Marketing Best Practice: Communication Strategy https://econsultancy.com/reports/email-marketing-communication-strategy/

Econsultancy | Email Marketing Best Practice: Copywriting

https://econsultancy.com/reports/email-marketing-copywriting/

Econsultancy | Email Marketing Best Practice: Layout and Creative

https://econsultancy.com/reports/email-marketing-layout-creative/

Econsultancy | Email Marketing Best Practice: List Quality and Deliverability https://econsultancy.com/reports/email-marketing-list-quality-deliverability/

7.2 Other Econsultancy reports

Econsultancy + Adobe | 2022 Digital Trends: Experience Index

https://econsultancy.com/reports/2022-adobe-digital-trends-experience-index/

Econsultancy | Content Strategy Best Practice Guide

https://econsultancy.com/reports/content-strategy-best-practice-guide/

Econsultancy | The Fundamentals of Market Research and Insights

https://econsultancy.com/reports/fundamentals-market-research-insights/

7.3 Other reports

RPE Origin | AI in Email Marketing Research Report

https://www.rpeorigin.com/2023-ai-in-email-marketing-research-report/

DMA | Marketer Email Tracker 2021

https://dma.org.uk/research/marketer-email-tracker-2021