Reading results

Planned rate vs real rate

A split test is planned from a guess of your KPI's rate. Once real numbers come in, Needlr redoes the plan if the guess was far off.

In Needlr: Check page, the progress bar and What counts as success

0:00 / 1:04

Every split test starts with a plan: how many visitors each version needs. That plan rests on one input, today's KPI rate, and at setup that's often a guess.

A split test is planned before it starts: how many visitors each version needs. That plan uses how often your KPI happens today. If that number was a guess, Needlr checks it against the real one and fixes the plan when it's far off.

Think of a road trip

You plan a trip: 300 km at 100 km an hour, so 3 hours. After the first stretch you see the road is slower, more like 50 km an hour. A good driver updates the arrival time to 6 hours. They don't stop the car after 3 hours and say “we're there.”

Needlr is that driver. The trip is the planned number of visitors. The speed is your KPI's real rate.

Where the guess comes from

When you create an impact check, step 3 (How should we test it?) has Advanced settings. There, Today's KPI rate is the rate the plan starts from.

If Needlr already counts this KPI on your site, it fills the box in and says Measured from your site.
If not, it starts from 3% and says A guess is fine; Needlr measures the real one. You can type your own number.

When Needlr re-plans

First, version A (your current page) needs real data: at least 2 days, 300 visitors and 10 conversions.
Then Needlr works out A's real rate. It uses the visitors of both versions to do it, so one lucky day on A doesn't move the plan.
If the real rate is less than about two thirds of the planned rate, or more than one and a half times it, Needlr redoes the plan from the real rate.
If the real rate is close to the plan, the original plan stays.

The new plan keeps everything else you chose: the Smallest improvement worth knowing and how sure the verdict must be. Needlr checks again at every update, every 5 minutes, so the plan settles as the rate settles.

An example: you planned with 3%, looking for +20%, 95% sure. That's 13,914 visitors per version. The page really converts at 1.5%, so the new plan is 28,304 per version: about twice as many. If it really converted at 6%, the plan would drop to 6,719.

What you'll see

The progress bar says, for example, Planned: 28,304 visitors per version · updated to your real rate (1.5%).
The percent done and the days left change. A bigger plan means a lower percent and more days. A smaller plan means the opposite.
The verdict date on the timeline moves, and so does the date on Add the verdict date to my calendar.
What counts as success shows Real rate instead of Assumed at setup. Before a re-plan, the Assumed at setup card already shows version A's rate so far under it, so you can compare.
The needle's scale follows the real rate, so the dashed marks still show the smallest change this check can notice (the needle).

What counts as success

Real rate1.5%

Detects changes of20% or more

Verdict confidence95%

Real rate: of the current version, estimated from both

After a re-plan, the card shows the real rate instead of the one assumed at setup.

Why it matters

Guessed too high: the old plan is too small. The check would end before it could notice a real win. You'd get No clear effect for a change that worked.
Guessed too low: the old plan is too big. You'd wait weeks longer than you need to.
Either way, how sure the verdict must be stays the same. Re-planning doesn't make a false winner more likely.

This is only for split tests. A Before vs after check has no visitor plan: it compares the 28 days after the release with the days before (more).

TipA re-plan can make a check much longer. If the new date is too far away, see Too slow to finish.

Common mistakes

Spending ages on the perfect guess. A rough number is fine. Needlr corrects it.
Stopping a check because the days left went up. That's the plan getting honest. Stopping throws the answer away (more).
Expecting a re-plan for a small miss. 3% planned and 2.5% real is close enough. The plan only changes when the guess was far off.

Words to know

Planned rate: The KPI rate the plan started from: measured from your site, or a guess.
Real rate: Your current version's rate, worked out from the check's first days.
Planned sample: How many visitors each version needs before Needlr gives a verdict.
Re-plan: Needlr working out the planned sample again from the real rate.

What the video says

Every split test starts with a plan: how many visitors each version needs. That plan rests on one input, today's KPI rate, and at setup that's often a guess.

Once version A has two days of data, 300 visitors and 10 conversions, Needlr measures the real rate. Here you guessed 3%, but the page actually converts at 1.5%.

When the real rate is far from the plan, below two thirds or above one and a half times, Needlr redoes the plan. At 1.5%, it needs twice the visitors.

On the check page, the progress bar reads updated to your real rate, the days left and the verdict date move, and the needle's scale adjusts to match.

A rosy guess would end the test too early and miss real wins. A pessimistic one would waste weeks. Either way, the confidence you chose stays the same.

So a rough guess at setup is fine. Needlr checks it against reality and keeps the test sized to find the lift you care about, at the confidence you picked.