A strong A/B testing plan will allow you to significantly improve or enhance your KPIs. The deep and accurate knowledge you gain regarding your customers’ preferences and perceptions of your offering, is worth its weight in gold.
A strong A/B testing plan will allow you to significantly improve or enhance your KPIs. The deep and accurate knowledge you gain regarding your customers’ preferences and perceptions of your offering, using a well thought out A/B test, is worth its weight in gold. This information will make product, marketing and other strategic decision-making much more efficient and sound.
A/B testing produces concrete evidence regarding what actually works in the product or service you are offering. In addition to product and marketing insights, continuously testing your hypotheses will also yield good conversion rates.
In its core, A/B testing belongs to a category of Scientific Optimization techniques, where statistical tools and models are used to increase the odds that your site visitors will see the best-performing variant.
Scientific optimization can be broken down into three categories:
A/B Split Testing
Simple testing of one element against another to see which one performs better.
Multivariate Testing
Testing several elements at a time.
Custom Experimental Design
Developing your own research method for an in-depth analysis.
In this post I will focus on A/B Split testing, since this technique produces the fastest gains and has lower chances of error through misuse, plus being usually very interpretable to other people in the organization.
Commitment to A/B Testing
Well thought out A/B testing requires an investment of time and resources across the organization, the extent of which depends on the available A/B testing infrastructure. For A/B testing to produce maximum improvement of all customer interactions, it is important to optimize all funnels. Before setting out, it is crucial that all departments in the company are committed to the A/B testing process, in order for it to be as effective as possible.
The A/B Test Process
Ask a Question
ex. “How can I get more users to register to the app?”
Perform Background Research
Use Events or web analytics such as Google Analytics, to track user behavior.
Construct a Hypothesis
ex. “Removing Facebook connect window upon on-boarding will increase registration completion”
Design A/B test that checks the hypothesis
ex. Create an A/B test where the variant version is an exact copy of the existing interface, minus the FB connect window.
Analyze Data and Draw Conclusion
ex. If the variant without the FB connect window has a statistically significant higher completion rate than your control, you can conclude that removing the FB connect window will, in all probability, increase registration completion.
Communicate Results
Let other stakeholders in your company know about your findings. This will allow everyone to be on the same page. The advantages to this are:
Ensuring that other departments are not wasting valuable time looking into something you have already researched.
Providing your colleagues with ideas for A/B testing methodology and applications, which are perhaps relevant to their domains as well.Your results may be picked up and run with by other departments, for further research.
List of Common Best Practice A/B test types you may consider:
Headlines
Color/Size of Buttons
Calls to Action
Homepage Design
Social Sharing Widgets
Images
Copywriting
Sign up Page & Onboarding Funnel
Media Quotes.
New Features and Updates
Offering Details
Importance of Choosing the A/B testing Framework that is Right for You
A/B testing needs vary across companies and products. . It is essential to map out and understand one’s requirements prior to purchasing any SaaS solution or endeavoring to build your own A/B testing system. On the one hand, building your own system is bound to have the flexibility and capabilities you are looking for, but on the other, this will possibly come at a very high cost. I would advise starting with the MVP and building upon it gradually. Using a service will buy you rapid time-to-market, but will not always give you exactly what you need; i.e. some configuration and coding work may well be required in order for tests to run in maximum efficiency.
For example: if the success metric is of a kind that the Saas solution cannot work with, it will either require significant modifications for it to work, or this metric will be abandoned for the time being. In cases such as these, companies tend to opt for building their own system, subject to their budgetary limitations.
Don’ts of A/B Testing
Don’t start testing without a data-driven, learning-oriented hypothesis.
Don’t come into it with preconceived ideas. You don’t know until you check and may often be surprised by which variant performs better.
Don’t rush testing. Time is an important parameter that should not be taken lightly. Every test should have a designated duration, depending on traffic and other considerations. As a rule of thumb, I suggest running each test for at least two weeks.
Don’t test partial weeks - always test full weeks at a time.
Don’t give up after the first test, if your hypothesis fails. Test on an ongoing basis to improve results. On the other hand, if your hypothesis fails consistently, consider reviewing it.
Don’t test low impact changes. Focus on where the most impact can be achieved (ex. Top bounce rate pages, critical points in your funnel, shop bounce rate etc.)
In Summary
A/B testing technique consists of statistical methods and analysis that will improve product KPIs, provided that the company is committed to sound A/B testing methodology and data driven decision making. That being said, you do not need to be a statistician in order to understand the concept and significance of your A/B tests, once the infrastructure is in place and the key metrics for success have been laid out.
Before setting out, one has to decide on the testing infrastructure that will enable optimal testing of the product or service. A website, an app, an IOT device and a wearable device all have uniquely different testing profiles and requirements that need to be met by your selected infrastructure.
While there is some expertise to be gained in conducting A/B testing, conducting a well-considered and well executed test sequence, can provide invaluable insight for any company wishing to optimize its products, services, marketing and overall decision-making.
(Many thanks to Micky Daniels for his linguistic and editorial advice)
Read more about:
BlogsYou May Also Like
