Product managers like Kerstin Exner are encouraged to run A/B tests whenever possible, as long as a strong business case can be made for each experiment.
One property The Guardian optimizes frequently is its dating site, Soulmates.
The original navigation bar displayed six default tabs.
Because a site visitor’s “Top matches” do not change frequently, Kerstin hypothesized that visitors thought the site looked stale and lifeless.
The negative and insignificant results helped her plan for future tests by honing in on new ways to increase conversions.
Setting a hypothesis before building each test makes executing and interpreting experiment results much easier.
Additional user research around Kerstin’s “people first” hypothesis revealed that many site visitors craved more information on Soulmates’ homepage – which displayed photos and quirky taglines for several single men and women on the site.
Kerstin pitted the original homepage carousel against one that replaced the quirky taglines with the age and location of each featured user.
According to Kerstin, agreeing on clear goals upfront helps remove the debate aspect when her team looks at test results and also influences ideas for future experiments.
“Of the 14 tests we have run so far on Soulmates, seven have been successful, three have been negative, and four have been inconclusive, in that the differences did not generate big enough changes to reach statistical significance.
With more than 6 million weekly readers, The Guardian is the third most popular newspaper globally.
The paper’s strength lies in its strong “digital-first” approach and data-driven culture.
They have been testing with Optimizely since early 2012 to grow online readership.