3 min read

A/B Testing: The Secret Weapon of Growth

A/B testing replaces guesswork with evidence. Learn how to run your first test, read the results honestly, and grow your conversions.

Cover image for A/B Testing: The Secret Weapon of Growth

A/B testing means showing two versions of a page to different visitors and keeping the one that performs better. That's it. No redesign debates, no "I think users prefer blue", just evidence. If you change your website based on hunches, you're gambling with traffic you paid for.

Why guessing costs you money

Most website decisions get made the same way: someone senior has an opinion, the team nods, the change ships. Nobody ever finds out whether it helped or hurt.

Here's the uncomfortable part: even experts guess badly. Microsoft's Bing team once deprioritized a tiny ad headline tweak for months because nobody believed in it. When an engineer finally tested it, revenue jumped 12 percent, worth over 100 million dollars a year in the US alone, according to a Harvard Business Review article by Ron Kohavi and Stefan Thomke. The idea everyone ignored was the biggest win in years.

If Bing's own specialists couldn't spot their best idea by intuition, your gut isn't going to fare better. Testing is how you find out what actually works on your site, with your visitors.

What can a single test change?

Sometimes a lot. In a VWO case study, backup software company Veeam changed one link label from "Request a quote" to "Request pricing". Clicks to the sales form went up 161 percent. Same page, same traffic, three words.

That's an outlier, not a promise. Most tests move the needle a little or not at all. But small, compounding wins are exactly how the big players grow: Harvard Business School professor Stefan Thomke documented that Booking.com runs more than 1,000 experiments at any given moment, roughly 25,000 a year. They didn't become a conversion machine through one brilliant redesign. They earned it one test at a time.

How do you run an A/B test properly?

  1. Start with a real problem. Look at your analytics first. Where do people drop off? A test without a hypothesis is just redecorating.
  2. Change one thing at a time. Test the headline, or the CTA, or the form length. If you change all three and conversions rise, you've learned nothing about why.
  3. Wait for statistical significance. Early results lie. A variant can look like a winner on Tuesday and a loser by Friday. Let the test collect enough visitors before you call it, and decide the sample size up front.
  4. Test where the traffic is. A test on a page with 50 visitors a month will take a year to conclude. Start on your homepage, top landing pages, or checkout.
  5. Keep a log. Write down what you tested, what you expected, and what happened. Losing tests are tuition, not waste: they tell you what your visitors don't care about.

What should you test first?

  • Headlines. The first thing visitors read, and the cheapest thing to change.
  • Calls to action. Wording, placement, and contrast. We wrote a whole post on where to place your CTAs.
  • Forms. Try removing a field. Then another one.
  • Social proof. Testimonials near the decision point versus buried on a separate page.
  • Page layout. If a page feels heavy, test a simpler version. Clutter has a real cost: see our post on cognitive load and conversions.

The mindset that makes it work

A/B testing isn't a tactic you deploy once. It's an admission that you don't know what your visitors want until they show you. The companies that win at this treat every strong opinion as a hypothesis waiting for data.

So next time someone on your team says "users will love this", smile and say: "Great. Let's test it."

FAQ

How long should an A/B test run?

Until it reaches statistical significance, and at least one or two full business cycles (usually two to four weeks) so weekday and weekend behavior both show up. Stopping early is the most common way to get a false winner.

How much traffic do I need to A/B test?

As a rough rule, a page needs thousands of visitors and dozens of conversions per variant to produce a trustworthy result. Low-traffic sites do better testing big, bold changes rather than button shades.

What's the difference between A/B testing and multivariate testing?

A/B testing compares two versions of a page. Multivariate testing compares combinations of several elements at once, which needs far more traffic. Start with A/B.

← All articles

Want this applied to your site?

Get a free teardown. We'll point at the exact sections costing you conversions.