A/B testing and experimentation

A/B testing lets a business make decisions from data, improve its marketing continuously, personalize experiences and improve the user experience. Done well, it leads to better results, happier customers and a higher return on investment.

What I believe about testing

• Decisions should come from data. Testing shows how real users respond to different versions of a page, email or ad, and that evidence should decide what you ship.
• Optimization needs a controlled comparison. Testing headlines, images, calls to action and layouts against each other identifies the version that actually performs best.
• Improvement is continuous. A testing culture keeps refining strategy based on real-world data instead of assumptions or gut feeling.
• Personalization should be tested too. Variations tailored to segments of your audience lead to more relevant experiences, higher engagement and more conversions.
• Testing is cost-effective. It finds problems and improvements before you commit budget, so resources go to what works.
• The user experience matters. Navigation, content layout, forms and checkout can all be tested to make the experience smoother.

Testing methods I use

• Multivariate testing (MVT). Tests several elements at once, such as combinations of headline, image and call to action, to find the best combination.
• Sequential testing. Tests variations one after another, to see how changes perform over time and at different stages of a campaign or journey.
• Bandit testing. Algorithms shift traffic toward the winning variation as results come in, so less traffic is wasted on poor performers.
• Personalization testing. Serves tailored experiences to different segments and tests which variation works best for each.
• Bayesian A/B testing. Updates prior knowledge with experiment results as data arrives, which converges faster and handles smaller samples well.
• Contextual bandit testing. Adapts variations to each user’s context, such as location, device or browsing behaviour.
• Advanced analytics. Funnel, cohort and segment analysis show how each variation changes behaviour across the whole journey, not just the conversion rate.

Experience

• Strong knowledge of A/B testing, measurement and the statistics behind them
• Testing platforms including Adobe Target and Google Optimize
• A/B and multivariate tests, Auto-Allocate, Auto-Target, Automated Personalization, Target Recommendations and machine learning
• Steward of the testing process: gathering ideas and requirements and prioritizing what to test
• Project-managing whole testing programs: methodology and documentation, test setup and QA, execution, custom analysis, and reporting results
• Brainstorming test ideas with stakeholders, writing hypotheses, setting success metrics and choosing the right test design
• Sharing results and recommendations with stakeholders and senior leaders on a regular cadence, as data stories that make sense at the most senior levels
• Strong HTML, CSS and JavaScript, with development experience
• Coordinating many simultaneous tests so they don’t conflict, and monitoring them while they run
• Managing multi-site functionality, tracking and tagging infrastructure
• Validating sample sizes and variant splits before launch
• Researching business trends and customer behaviour to find opportunities for site improvements
• Applying statistical analysis to business problems

Always be testing

Paid traffic is expensive; increasing your conversion rate often isn’t. The return on A/B testing can be huge, because small changes to a landing page can produce significant increases in leads, sales and revenue.

For example: testing ad copy shows which version attracts more clicks, and testing the landing page shows which layout converts best. When every step works as efficiently as possible, the overall cost of acquiring customers goes down.

The best part of testing is that it lets you form hypotheses and learn why elements such as the headline, imagery, form fields, call to action and layout change user behaviour, through data.

Build the test plan around the elements likely to produce significant results. A banner or hero image will usually move conversions more than a change in the footer, and including less influential elements only increases the traffic and time a test needs.

What I do

• Work with product, business and customer journey teams to create and inform a global testing roadmap
• Find effective A/B and multivariate test opportunities, develop test plans with stakeholders, validate setup and measure results, including which segment-level data matters to each stakeholder
• Define test criteria, methodology, metrics and design, then build, run and make recommendations from each test
• Use the relative influence of each element in a full-factorial multivariate test to find the element with the most impact, then A/B test it. I have used this to test text and visual elements together, CTA text and colour together, and number of form fields with CTA text.
• Maintain timelines and testing processes, and improve them
• Treat every test as a lesson. There are no losers in A/B testing: the data from each test feeds the next iteration.
• Champion a test-and-learn culture, and maintain a global process to govern and educate business teams on testing and personalization
• Use analysis to find the segments that emerge from personalization and testing
• Manage personalization platforms such as Adobe Target to deliver unique experiences to individual customers and segments
• Work with external testing partners and internal teams such as merchandising, UX, creative, editorial and technology
• Balance testing methodology with each stakeholder’s business rules
• Present results clearly and quickly, with actionable recommendations for every level of the organization

To size tests properly I also use Python for power analysis and Bayesian statistics. See Python for analytics and A/B testing.