One of the common challenges marketers face is understanding what activity is having the biggest effect on changing customer behaviour.
However, with the plethora of channels out there and the number of interactions a customer has with a brand, is it truly possible to understand what has and has not had an effect? In the field of customer experience, some companies are using Multivariate Testing or (MVT) as an optimisation tool, while others are looking at specific A/B testing plans. Both of these options work well but when looking at marketing, where do you start?
Motor racing teams face a similar problem. The big challenge they are trying to solve is get the car to its optimum state to set the fastest lap time. They too face a huge number of variables, including the tyres on the car, tyre pressures, track temperature and the weather. That’s without the different driving styles and abilities of the drivers.
Starting Off On Track
The first thing the racing team will do is send the car out with a neutral or balanced setup, or use a previous setting. This setup is not fine-tuned. And for a new team it is like having the car with everything set to 0.
What the team will do is take a note of the lap time as this is one metric that allows them to understand if the car is improving. The driver might say the car is wild but if the stopwatch shows it’s faster there is no arguing!
Once they have this benchmark they will then start to make changes. These ideally will be small and one-off changes based on feedback from the driver and data logged by the car. If the team changed the gear ratio, the brake balance, the downforce and the suspension setting they wouldn’t know what made the difference.
The driver will then be sent out and the time benchmarked against previous performance. If the time is faster then the team are going in the right direction. They can then make the previous change bigger or start to change something else to complement the previous change.
This will take time and is very focused on little and often.
The important thing to remember is making a single change, measure and review its effect.
Getting Into Marketing Gear
The marketing team will have existing data from previous campaigns and digital activity. Looking at it in its entirety however can be overwhelming. However if you break down the customer journey by touchpoint this becomes far more manageable. In much in the same way the racing team will only focus on one car, one driver, one track and one type of weather at one moment in time.
Let’s look at an ecommerce business.
They will identify a number of customer phases, for example:
research and acquisition
returns and customer service
renewals and restocking (depending on the product)
They will also have a number of channels:
– Mobile site
In this example we will look at the customer research and acquisition phase and focus purely on the website and email channels. Our metric of success is new customers or online orders. It is important for us to identify the metric of success as this will be used to benchmark whether the activity has improved our existing number of customer orders or visit-to-purchase rate (conversion rate).
We want to be able optimise the acquisition part of the customer journey and therefore are going to perform A/B tests on the email and website channels.
As we are focused on acquisition on the website we can look at the conversion funnel and see where most of the customers are dropping out. This will inform us what stage customers are getting to, which customers are just visiting, and which ones have gone onto purchase from us.
For the email channel we can look at which abandoned basket triggered messages are being opened.
From here we want to be able to set out a number of hypothesises to test. These will form the basis of the “A test” (the new or optimised experience) vs. the “B test” (the experience everyone else will get.) Note these hypotheses should not overlap. For example we do not want to test the subject line, the time to the email and the address the email is sent from at the same time, as we will not know which change is affecting customer behaviour.
However we could change the location of the checkout button combined with the subject line on the triggered email as these would effectively be testing different things.
Once we have run this test with a our control group and a big enough set of customers, we can then review what has and has not worked. With larger audiences it is possible to run multiple versions of the same test for example we could split the A test into further creative tests. We could test a subject line with a 50% off offer and one with a Free delivery offer to see which one worked the best.
In the example above we can look to see the effect of the subject line. If it has been successful we can move onto the next test in the same channel and part of the customer journey. If it has had little or no effect then we can revise the subject line test.
Alternatively we can test the next part of the customer journey or another channel.
Getting Off To A Flying Start
So if we want to get the best results from campaigns and approach the problem like a racing team we need to:
Define the hypothesis we want to test against
– Set our benchmark
– Identify the variables (customer journey and channels) to work with, remembering less is more
– Isolate the test to ensure we are not clouding the results
– Make changes little but often
– Review the outcome and move onto the next test
his might seem like a simple list but once the journeys have been mapped out by testing in a methodical way any improvement – no matter how small – is understood, rather than trying to work out what test worked only to have tested nothing at all.
This article was originally published on Cmo.com
Thank you to the British Library for their getting into gear image available from their flickr photostream.