A/B testing, what about killing assumptions?
Mixed feelings about this article on A/B testing.
I agree it’s easy to fall into the trap of A/B testing and test every single detail, but that’s not necessarily what A/B testing was made for.
Think about these never-ending design meetings, the kind of meeting every participant knows exactly what to do, what the end-user wants and what will make the website a success.
Or, think about a client who “knows” how to design efficient websites.
A/B testing (or multivariate testing) is just the perfect way to kill assumptions. You might not increase your conversion rate by a significant margin, but you’ll surely avoid endless arguments about the best layout, colours, image choice, etc. Statistics will tell (or prove everyone right because no design change actually makes a difference).
The Bad: Relying Too Heavily On Perfection
Not everyone agrees, however, that A/B testing is a wise practice. Just this morning, Jeff Atwood, author of the blog Coding Horror, expressed his idea that this kind of testing is a lot like the movie Groundhog Day. In the movie, Bill Murray - stuck in an infinite loop reliving the same day over and over again - attempts to win the love of a woman, Rita, by learning as much as he can about her each day and pretending to share all of her interests.
Atwood says that this is exactly what startups and designers are doing by relying too heavily on A/B testing. While romantic relationships are certainly different than business relationships, the parallels are interesting. Even though Murray’s character says presumably “all the rights things,” Rita can still tell there is something odd about it. Startups should be wary of testing too many of their decisions and should instead focus on just a few key elements at a time.
Rand Fishkin at SEOmoz warned a few weeks ago against falling into the “trap of A/B testing minutiae.” He says many are “tantalized” by the idea that a small change can go a long way, when in reality this is far from the norm.
“In all of these, some simple change accounted for big increases in click-through or conversion rate, leading to widespread praise and sharing,” says Fishkin. “The problem is - they’re the exception, not the rule. In fact, that’s precisely why they’re newsworthy and get so many mentions.”
Fishkin also says for the small changes A/B testing often produces, the time and energy is too high. He suggests that startups focus more on testing larger changes - like a sight redesign - rather than smaller ones - like button colors.
Read more at www.readwriteweb.com
