A/B Testing
A/B testing proves whether a change actually helped. It only does that when there is enough traffic to separate a real effect from noise, and most B2B sites do not have it on the pages that matter.
What a test actually settles
Whether version B outperformed version A by more than chance would explain. That is a narrower claim than it sounds, and it is the whole value: without it, a change that coincided with a good month gets credited for the month.
How much traffic does a test need?
More than most people expect, and the requirement grows sharply as the effect you are trying to detect gets smaller. Detecting a large improvement takes relatively few conversions. Detecting a modest one takes a great many.
The practical threshold is conversions, not visits. A page with heavy traffic and two enquiries a month cannot test, because the thing being measured barely happens.
Work it out before running anything: current conversion rate, the smallest improvement worth acting on, and how long that many conversions will take to accumulate. If the answer is nine months, the test is not viable and pretending otherwise wastes the nine months.
What happens if you test anyway
You get a number, it looks decisive, and it is noise. Then it gets applied to the rest of the site as a finding.
That is worse than not testing. Not testing leaves you uncertain, which is honest. An underpowered test leaves you confident and wrong, and the error propagates into every page built on it afterwards.
Peeking manufactures wins that do not exist
Checking a running test repeatedly and stopping when it looks good. It is the most common way to record a win that does not exist.
Results fluctuate throughout a test, so a variant will cross into apparent significance by chance at some point. If you only stop when you like the number, you will stop on noise almost every time. Set the duration in advance and let it run.
How long a test should run
At least one full business cycle, and for B2B that means whole weeks rather than days. Traffic on a Tuesday behaves differently from traffic on a Saturday, and a test that ran Monday to Thursday has measured weekdays, not your audience.
Where the test involves how something is retrieved or recommended rather than just clicked, allow longer still. Systems that surface content adjust slowly, and a few days tells you nothing about where a change settles.
When the site cannot support testing
Most of what actually improves conversion does not require a test, and this is the honest answer for the majority of B2B sites.
| Instead of testing | What it gives you |
|---|---|
| Fix the measurement first | Numbers that mean something. Frequently changes the picture before any page is touched |
| Watch real sessions | Specific obstacles, observed rather than hypothesised |
| Form and field analytics | Exactly where people abandon, which is usually one field |
| Careful before and after | A directional read, with the seasonality caveat stated rather than hidden |
| Fix the obviously broken | A page that is slow, unreadable on a phone or asks for a budget range does not need a test to justify repair |
What a worthwhile test looks like
- A hypothesis, written down first. What you expect to change, and why. "Let us try a different button colour" is not one.
- One variable. Change the headline and the form and the layout together, and a win tells you nothing about which part won.
- Sample size and duration fixed in advance, along with what result would count as a failure.
- The losing result reported too. A disproved assumption is a real outcome and stops it being retried next year.
Where this fits
Inside conversion optimization, after measurement is trustworthy and usually after behaviour analysis has produced something worth testing. Tests on landing pages fed by paid traffic are often the only ones with the volume to be viable.
Frequently asked questions
Can we test on a page with a few hundred visits a month?
Not meaningfully. We will say so rather than take the work. That page can still be improved through observation and repair, which is where the budget should go.
What if the test shows no difference?
That is a result. It means the thing you changed was not what was holding the page back, which narrows where to look next. Tests that fail are only wasted if nobody records them.
Does testing hurt SEO?
Not when implemented properly. Serving different versions to real users is expected behaviour and search engines account for it. Problems come from leaving test infrastructure running for years or from cloaking, which is a different thing wearing a test's clothes.
See where you stand
We will review your current search visibility and show you what buyers find when they look for what you do.
Contact Us