How big and how long

How many visitors a split test needs, how long that takes, and the three settings that change it.

Before a split test starts, Needlr works out how many visitors it needs, and how long that will take. Three settings in step 3 decide it.

Think of a coin

Is a coin fair? Flip it 4 times and get 3 heads: that proves nothing. Flip it 1,000 times and get 750 heads: now you know something's up.

A split test works the same way. A big difference shows up quickly. A small difference needs many, many visitors before you can trust it.

How many visitors see the new version?

A slider from 1% to 99%. That share of your visitors sees the new version. The rest see your site as it is.

• 50%, half and half, gives you the fastest answer.
• Less than half is safer if your idea turns out bad, because fewer people see it. But it's slower: at 10%, the impact check takes about five times as long.
• More than half is slower too, because now the group on your current site is the small one.

Why? The smaller group decides how long it takes. Both groups need enough visitors, and a small group fills up slowly.

How sure do you want to be?

A slider from 70% to 95%. It's how sure Needlr must be before it says Improved or Got worse. Lower means a faster answer, but a wrong answer more often.

• 95% (very sure, the default): about 1 answer in 20 could be luck. The safest and the slowest. Use it for prices and big changes.
• 90% (very sure): about 1 in 10 could be luck. Good for most changes. Needs about a fifth less data.
• 80% (fairly sure): about 1 in 5 could be luck. Fine for page changes, not for prices. Needs a bit more than half the data.
• 70% (quick read): about 3 in 10 could be luck. Only for small things, like a button's words. Needs a bit less than half the data.

The level is fixed once the impact check starts. A verdict reached below 95% says so next to it.

Advanced settings

Press Advanced settings under the sliders to see two more:

• Today's KPI rate: how often your KPI happens now, in %. Needlr fills it in from your site when it can. A guess is fine.
• Smallest improvement worth knowing: +10%, +20% (the default), +30% or +50%. It's the smallest lift you'd care about. Needlr shows how many visitors each group needs for it.

This one changes the wait the most. Looking for half the lift needs about four times the visitors.

Your guess for today's rate doesn't have to be right. After 2 days, once there are at least 300 visitors and 10 conversions on your current version, Needlr checks the real rate. If it's far from the guess, Needlr redoes the plan from the real rate, and the check page says so.

How long will it take?

The What happens box in step 3 tells you, for example First answer in about 6 weeks. It's worked out from your visitors per week and the size of the smaller group.

An example: a page with 5,000 visitors a week, a KPI at 3%, half and half, 95% sure.

• Looking for +20%: about 13,900 visitors per group. About 6 weeks.
• Looking for +50%: about 2,500 per group. About 2 weeks.
• Looking for +10%: about 53,000 per group. More than 2 months.

With the planned number of visitors, a real improvement of the size you picked is found 8 times out of 10. And a split test never gets a verdict before 7 full days, even on a busy site (why).

Too slow? What to do

• Test a bigger, bolder change. Big changes show up with fewer visitors.
• Test on a busier page.
• Pick a KPI that happens more often, like a click on the page itself instead of a purchase three steps later (how).
• Keep the split at half and half.
• For small, low-risk changes only, lower how sure you need to be.

If a running impact check is so slow it would need more than about 2 months, the check page warns you: At this pace this check won't reach a verdict.

Common mistakes

• Choosing 70% for a price change. About 3 answers in 10 could be luck. Keep 95% for anything that costs money if it's wrong.
• Hoping to find +10% on a quiet page. It may take months. Test a bolder idea.
• Thinking more visitors on the new version is faster. It isn't. Half and half is the fastest.

Words to know

Traffic share

How many of your visitors see the new version.

Confidence

How sure the verdict must be. 95% means about 1 answer in 20 could be luck.

Smallest improvement

The smallest lift worth finding. Smaller needs many more visitors.

Planned sample

How many visitors each version needs before Needlr gives a verdict.

KPI rate

How often your KPI happens: out of 100 visitors, how many do it.