What the verdicts mean
Improved, Got worse, No clear effect: what each one means, and what to do next.
When all the planned data is in, Needlr makes one call and freezes it. Later visitors don't change it, so nobody can peek their way to a winner.
A verdict is your impact check's answer to one question: did your change help, hurt, or do nothing? There are three answers. Before them come two “not yet” stages, and one “something is wrong” stage.
Think of a referee
A referee doesn't end the game at half time because one team is ahead. They wait for the final whistle. Then it's a win, a loss, or a draw.
Needlr works the same way. Improved is a win for your new version. Got worse is a loss. No clear effect is a draw.
When does the verdict come?
- Split test: when each version has had its planned number of visitors, and at least a full week has passed. Weekdays and weekends bring different visitors, so a split test always covers a whole week.
- Before vs after: after 28 days of data from after the release.
- Adoption: at the end of its window.
Once a split test has its verdict, it's final. Later visitors don't change it. The top of the page says Final · decided and the date. Owners and Editors also get an email with the result.
What each one means
Not enough visitors yet. The first numbers show up after about 100 visitors per version and 10 KPI hits.
Numbers are coming in, but not enough yet. You may see Leaning up or Leaning down. That's a hint, not a result. Don't act on it.
The new version (B) got more people to do your KPI. The whole range is above zero.
The new version made your KPI go down. The whole range is below zero.
No real difference. If there is one, it's smaller than this impact check could notice.
Something is wrong with the counting, so the impact check is on hold and won't decide (what to fix).
What to do next (split tests)
- Improved: press Ship version B at the top, so every visitor gets it now (how). Or press Copy a note for your developer and send it.
- Got worse: press Stop the check. Everyone goes back to the current version. The page tells you about how many KPI hits B costs you each week.
- No clear effect: keep whichever version is simpler to look after, and press Stop the check. Then try a bigger idea.
Before vs after, and Adoption
A Before vs after impact check has no group that kept the old version, so other things that changed in the same weeks can play a part. Its answers say so: Likely helped. Keep it., Likely hurt. Consider rolling it back., or No clear change. (more)
An Adoption impact check compares how many people used your feature with your target: the target was reached, the share stayed below it, or it is too close to tell (more).
On the project page
Your project's list of impact checks uses the same words: Improved with the lift (like Improved +12%), Got worse, No clear effect, On hold, Too early to call and Collecting data. A split test that has its verdict but still splits traffic starts with Verdict ready. Adoption impact checks show Reached, Below target or Near target.
How sure is a verdict?
Most impact checks need 95% confidence. That means a verdict is wrong about 1 time in 20. Some are set lower when they're created, to finish faster. Then the verdict says so, for example “At 80% confidence.”, and it is wrong more often.
Common mistakes
- Treating No clear effect as a failure. You learned this change doesn't matter much. That saves you from building more of it.
- Acting on Too early to call. Leans flip all the time (why).
- Running the same impact check again and again until it says Improved. Sooner or later luck says yes. Trust the first verdict.
Words to know
- Verdict
- Needlr's final answer for an impact check.
- Range
- Where the real difference most likely is. On the page it's the band around the needle.
- Lift
- How much better or worse the new version did, in percent.
- Confidence
- How sure Needlr must be before it calls it. 95% is the standard.
What the video says
When all the planned data is in, Needlr makes one call and freezes it. Later visitors don't change it, so nobody can peek their way to a winner.
Improved means the whole range sits above zero. Version B really lifted your KPI. Ship it to everyone, and claim the lift as a win your team delivered.
Got worse means the whole range sits below zero. The change is costing you conversions every day it runs. Stop it, and everyone goes back to the current version.
No clear effect means the range still crosses zero at the finish. Any real difference is smaller than this impact check could detect. Keep the simpler version, or test a bolder idea.
Before the call you'll see Collecting data, then Too early to call, with a lean that may say up or down. A lean is not a result. Don't act on it.
And if the data itself breaks, the impact check goes on hold as a Data problem instead of guessing. Every verdict comes with its next move: ship, roll back, or iterate.