How Needlr decides
The method behind every verdict: a fair split, a range, a planned sample, a full week, and one call that never changes.
You create an impact check to see if the needle moved. Before you act on the answer, someone will ask: how do you know? Here is the method behind every verdict.
Every verdict in Needlr follows the same few rules. This page explains them in plain words. The full version, with the math, is on the methodology page.
Think of a science fair
You want to know if a new plant food works. You take two plants of the same kind. They sit at the same window and get the same water. Only one gets the new food.
Before you start, you decide: “I'll measure them after three weeks.” You don't stop on day 2 because one looks a bit taller. Then you measure, and you write the result down. You don't change it later.
Needlr does the same with your website. The plants are the two versions of your page. The height is your KPI.
The five rules
- 1A fair split. In a split test, a coin flip decides who sees the current version (A) and who sees your change (B). Each visitor keeps their version. Both groups get the same days, ads and news. So if they end up different, your change did it.
- 2Only what happened after they saw it. A visitor counts once they actually saw their version. A KPI hit counts only if it came after that. Your team's own previews are never counted.
- 3A range, not one number. Needlr compares the share of visitors who did your KPI in A and in B. It works out a range where the real difference most likely is (more).
- 4Wait for the plan and a full week. Before a split test starts, Needlr works out how many visitors each version needs (more). It doesn't decide before that, and never before 7 full days. Monday visitors are not Saturday visitors (why).
- 5One call, then it's frozen. The verdict is decided once. Later visitors don't change it, even if the test keeps running.
How the range becomes a verdict
The whole range is above zero. Your change helped.
The whole range is below zero. Your change hurt.
The range includes zero. If there is a real effect, it's smaller than this impact check could see.
How sure is sure?
On step 3 of a new impact check, How sure do you want to be? sets the bar. 95% is the default. A lower level gives you an answer sooner, but it is wrong more often. A verdict reached below 95% always says so next to it.
When Needlr won't decide
- Too few visitors. Below 100 visitors per version or 10 KPI hits in total, the page says Collecting data.
- A broken split. If A and B get a different share of visitors than planned, something is wrong with the counting. The impact check goes on hold as a Data problem (more).
- A KPI that stopped firing. A broken button looks like a big drop. The impact check waits on hold until the KPI fires again.
- Unusual traffic. When one source sends far more visitors than normal, they are left out of the numbers (more).
- Proof for a change released to everyone. A Before vs after check has no group that kept the old version, so its answers say “likely”, not “proven”.
Common mistakes
- Acting on Too early to call. A lean is not a result. It flips often.
- Running the same change again until it says Improved. Sooner or later luck says yes. Trust the first verdict.
- Reading 95% as “95% better”. It's how sure Needlr is, not how big the change is. The size is the lift.
Words to know
- Verdict
- Needlr's final answer for an impact check: Improved, Got worse or No clear effect.
- Range
- Where the real difference most likely is. It's the band around the needle.
- Planned sample
- How many visitors each version needs before Needlr decides.
- Confidence
- How sure Needlr must be before it calls it. 95% is the standard.
What the video says
You create an impact check to see if the needle moved. Before you act on the answer, someone will ask: how do you know? Here is the method behind every verdict.
In a split test, visitors are assigned at random, and each one keeps their version. Both groups live through the same days and campaigns, so a difference beyond chance is your change.
Needlr counts each visitor once, from the moment they saw their version, and compares the share who converted. It works out a range for the real difference, not just one number.
Before a split test starts, Needlr sizes it from your KPI's usual rate and the smallest lift worth knowing. It waits for that sample and a full week, because early numbers swing.
Then Needlr makes one call and freezes it. If it kept recomputing, about 16% of do-nothing changes would show a winner or a loser at some point. Frozen, it stays at 5%.
And it refuses to guess. Too few visitors, a broken split or a KPI that stopped firing means no verdict. So when Needlr says improved, you can stand behind it.