The range

The band around the needle: where the real difference most likely is, and why a verdict needs all of it on one side of zero.

The range is the colored band around the needle on the check page. It shows where the real difference between your two versions most likely is.

Think of someone walking toward you

A friend walks toward you from far away. You guess how tall they are: “somewhere between 1.5 and 1.9 meters.” As they come closer, your guess gets tighter: “between 1.70 and 1.75.”

Needlr does the same with your KPI. With a few visitors, it can only say “the new version is somewhere between a bit worse and a lot better.” With many visitors, it can say “a little better, by about this much.” More visitors is like your friend walking closer.

Where you see it

• On the needle: the colored band around it. The middle line is zero: no difference (how to read the page).
• Under the big number, after a verdict: Most likely between two numbers, for example +0.2 points and +1.2 points.
• Before the verdict: if the band still covers zero, you see could still go either way under the lean.
• On the chart, The road to the verdict: the band over time, getting narrower as visitors arrive. The dashed shape on the right is where it's heading: a forecast, not a result.

Why it gets narrower

Every visitor adds a little information, so the band shrinks as visitors arrive. To make it half as wide, you need about four times as many visitors.

An example: version A gets 3.1% of visitors to book a demo. Version B gets 3.8%. That's a difference of 0.7 points.

• With 1,000 visitors on each version, the range is about −0.9 to +2.3 points. B could be worse, or much better. Nobody can tell yet.
• With 10,000 visitors on each, it's about +0.2 to +1.2 points. Now the whole range is above zero.

Why a verdict needs it on one side of zero

Zero means “no difference.” If the range still includes zero, “no difference at all” is still one of the likely answers. So Needlr can't say your change helped, or hurt.

Needlr only decides once all the planned visitors are in, and a split test also waits for a full week (why). Before that, a band that looks clear can still move. Once the verdict is in, it's final (verdicts).

It follows the check's confidence

When you create an impact check, step 3 asks How sure do you want to be?, from 70% to 95%. The range is drawn at that level.

• 95% (the standard): if you ran the same test 100 times, the range would catch the real difference about 95 times. About 1 verdict in 20 could be luck.
• 80%: with the same visitors, the range is narrower, so the check needs fewer visitors and finishes sooner. But about 1 verdict in 5 could be luck.

You can see the level on the check page: open What counts as success and look for Verdict confidence. A verdict reached below 95% says so, for example “(At 80% confidence.)”. The level can't change once the check has started (choosing it).

Points and percent

The range is in points: the gap between two rates. 3.8% minus 3.1% is 0.7 points.

The big number above it is the lift, in percent. 0.7 is about 23% of 3.1, so the lift is +23%. Both describe the same difference, in two ways.

Common mistakes

• Reporting only the big number. “+23%” sounds exact. It isn't. Share the range with it.
• Reading a band that crosses zero as “B lost.” It means “we can't tell.” B might be a little better or a little worse.
• Thinking a narrow band means a big win. Narrow means sure, not big. A tiny lift can have a very narrow band.

Words to know

Range

Where the real difference most likely is. On the page it's the band around the needle.

Points

The gap between two rates: 3.8% vs 3.1% is 0.7 points.

Lift

The same gap in percent of the old rate: +0.7 points on 3.1% is +23%.

Confidence

How sure the verdict must be. It sets how wide the range is drawn.