What is A/B testing
Also known as: split testing, A/B test, bucket testing, controlled experiment
A/B testing is an experiment in which two versions of something, such as a page, ad or email, are shown to randomly assigned groups of people to find out which produces more of the desired result. Version A is the current one; B is the alternative. Because the groups are random, the difference can be attributed to the change rather than chance, as long as the test has enough volume.
A quote request landing page had an eight field form. The team tested a four field version, splitting traffic evenly for two weeks.
Each version got 2,000 visits. The original produced 60 requests (3%); the new one, 90 (4.5%). With that volume, the difference is large enough not to be chance, and the short version becomes the default. The next test starts from there.
How to plan it
- Start with a hypothesis: what you think will improve and why.
- Change one thing at a time, so you know what caused the difference.
- Define the main metric in advance, such as quote requests or sales.
- Estimate how much traffic you need to detect the difference that matters.
- Run in full weeks, so Mondays are not mixed with weekends.
- Only stop once you hit the planned volume.
The mistake of stopping early
In the first days, it is common for one version to pull ahead and then lose. Checking the dashboard daily and stopping at the first lead ends up picking winners by chance. That is why volume and duration are set in advance, and the test is read only at the end.
Rigorous testing is an old idea. In 1923, American advertiser Claude Hopkins described coupon based tests to compare ads in his book Scientific Advertising, still without the statistical concepts used today.
Where to test
In Google Ads, custom experiments split budget and traffic between the original campaign and a test version. Google recommends a 50% split for the best comparison, and a cookie based split ensures each person sees only one version.
| What is usually tested | |
|---|---|
| Landing page | Headline, offer, proof, form length, button |
| Ads | Copy, images, audiences, bidding strategies |
| Subject line, sender, send time, call to action | |
| Pricing and plans | Prices and payment options, minding customer perception |
A/B, A/B/n and multivariate
An A/B test compares two versions. A/B/n compares several at once, which needs more traffic. Multivariate testing tries combinations of several elements together and requires a lot of volume; for most sites, sequential A/B tests give faster, clearer answers.
Common mistakes
- Testing irrelevant details, like color shades, before testing the offer.
- Changing several things at once.
- Stopping at the first lead.
- Running with too little traffic to reach any conclusion.
- Not logging tests, and repeating what was already tested.
Frequently asked questions
What is A/B testing?
An experiment in which two versions of a page, ad or email are shown to randomly split groups of people to find out which produces better results.
How long should an A/B test run?
Long enough to reach the planned volume, in full weeks. Stopping early, at the first lead, leads to wrong conclusions.
What should I test first?
What has the most impact: offer, headline, proof and form length. Small visual details usually change little.
Can I run A/B tests in Google Ads?
Yes. Custom experiments split traffic and budget between the original and test campaigns, and Google recommends an even split.
What is the difference between A/B and multivariate testing?
A/B compares two whole versions. Multivariate tests combinations of several elements at once and requires much more traffic.
Related terms
- Conversion rateConversion rate is the percentage of people who took the desired action out of everyone who had the chance...
- Landing pageA landing page is the page a person reaches after clicking an ad, email or link, built to lead to a single...
- CTAA CTA (call to action) is the element that invites a person to take the next step: a button such as "Get...
- CopywritingCopywriting is writing meant to lead readers to an action: buying, clicking, signing up or requesting...
- Marketing attributionMarketing attribution is how credit for a conversion is split among the channels and ads a customer...
- KPIA KPI (key performance indicator) is a number chosen to show whether a business is reaching a specific goal...
Rodrigo Fávaro
Founder of ROO3, a marketing and technology agency in São José do Rio Preto, Brazil. Builds AI products running in production (Tobia, gerar.app, Pense Mercado) and maintains the AI Benchmark, a public ranking of AI models.
X @rodmf LinkedIn rodrigofavaroMarketing you can measure, from the ad to the sale.
ROO3 builds and runs paid media, SEO and websites with real targets, dashboards and numbers. The first conversation is free.