What’s Testing?

We’ve all been in those meetings that get down into the nitty-gritty. Should this push notification include an exclamation point? Should this subject line be quick and concise or a little more descriptive? Is our website banner the right shade of blue?

If you asked a random person on the street one of these questions, they’d look at you like you’ve lost your mind. But for you and your fellow marketers, you know how crucial some of these questions can be. And you know that answering these questions correctly could make the difference when trying to craft a successful marketing campaign.

But wait—how do you know if you are correct? Do you just have to make a decision, say a quick prayer to the marketing gods, then launch your campaign?

No. Of course not. That would be scary. And probably end badly. So to avoid this risk and make sure they’re launching the best campaign, the best teams are constantly testing.

What do you mean by testing?

Good question. When we talk about testing, we mean launching and comparing different variants of your messaging campaign. These variants may differ in copy details like subject lines, aesthetic details like color or use of GIFs, or even in the offer, itself—and testing gives you the data you need to understand which variant resonates with your audience the most. This helps you understand which version of your campaign delivers on key engagement metrics most effectively before actually launching your campaign to your entire audience.

So what tests should I be running?

There are a few different types of tests that you should be aware of. Luckily they all have the same general goal: to give you a valuable sample of data about your audience before launching your campaign to the masses. Here’s the tests to know and the situations that call for them:

• A/B testing: This test luckily has a pretty intuitive name—as it tests an “A version” of your campaign versus a “B version” of your campaign. One of its strengths is its simplicity. It enables you to test one variable about your campaign. For instance, you might test an email subject line that says “A/B testing is effective and fun!” versus another line that says “I ❤️️ A/B testing!” Which drives the most engagement? (Our money’s on the heart emoji).
• Multivariate testing: You might be familiar with the word “multivariate” from your stats 101 class in college. But in case you slept through that class, multivariate essentially means “multiple variables.” So while an A/B test involves one variable changing, multivariate tests change multiple variables.

For instance, you might want to understand how changing your subject line and the CTA button in your email influences conversions. In this case, you’ll have four variants:

So while A/B testing gives you a yes or no answer about an isolated aspect of your campaign, multivariate tests compare different versions of your campaign on a more holistic level and study how different variables within your campaign interact.

But wait. Who’s seeing these variants of my campaign?

Another good question. If you’re sending these variants to all sorts of users, you’ll possibly launch a subpar version of your messaging to a significant group of users. That’s why it’s important to get just the right amount of data to inform your test results. But how much data is too much...or too little?

The general rule of thumb: 10%. You send Message A to 10% of your audience and Message B to another 10% of your audience. Then if Message A wins, that’s the message that gets sent to the other 80% of your audience. This is a rule of thumb for a reason, as these percentages typically yield a statistically significant “winner.” But not always. So it’s worth looking into more advanced methods that leverage all sorts of fancy statistical mumbo jumbo.

In general, A/B tests require less traffic to determine a statistically significant result. For a really high traffic campaign, it may be worthwhile to run a multivariate test, though know that it will require you to dip into a larger portion of your traffic in order to get a statistically significant result.

How often should I be testing?

Oh you know, only constantly. In reality, the teams that are consistently turning out successful campaigns are investing in ongoing testing, which supports constant iteration of your messages, campaigns and strategy.

So while testing to settle isolated debates is of course valuable, you should be testing regularly so that you always have data about your campaigns at your fingertips. Then there won’t be anything to debate. You’ll always have a fairly good idea of what’s working and what needs some more thought.

I’m ready to test! Now what?

Testing is easier than ever with more customer engagement solutions available. Braze Canvas, our customer journey tool, makes it easy for marketers to get a full view of their campaign all in one place. You can easily visualize the different steps, stages, and messages that go into your customer outreach. Such an interface also makes it easier to test, giving you the full view you need to workshop improvements on the fly. With tools like this, you and your team can quickly launch different versions of your campaign or site with little coding required. In most cases, testing is a stepwise process of:

• Creating your campaign. For example, an email campaign.
• Creating variants within this campaign. Maybe an email with a blue CTA button and an email with a red CTA button.
• Schedule your campaign. Why not run it tomorrow?
• Choose a segment to launch this campaign for. Typically a statistically significant portion of your target audience, which we covered above.
• Divide users up between your variants. Use the 10% rule of thumb, or another logic that you and your team agree on.
• Figure out which KPIs you’ll be tracking. Which action are you going to be looking for users to make after seeing this campaign? Conversions? Click-through rate? Both?
• Review, launch, view your results!

Test away!

Testing. It’s easy, it’s valuable, and it helps you get a little bit closer to understanding what goes on in the minds of your customers. So what are you waiting for? Craft some variants, determine your segments, and get some much-needed answers about customer engagement.