Agents' statistics
Split tests your coding agents run get two extra safety nets: they stop early when they clearly hurt, and a win that may be luck is marked Not confirmed.
Agents run test after test, and nobody watches each one. That changes the math. Needlr adds two safety nets to every split test an agent runs.
Split tests your coding agents run get two extra safety nets. They stop early if they clearly hurt. And a win that might be luck is marked Not confirmed.
Why agents need more
A person runs a few tests and watches each one. An agent can run many, one after another, and nobody watches each one.
At that pace, two things go wrong. A bad change keeps running and costing you. And some “wins” are just luck.
Net 1: stop on harm
Think of a smoke alarm. It doesn't wait until the end of the day to tell you the kitchen is on fire. But a good one doesn't go off every time you make toast.
Needlr checks each agent split test at every update, about every 5 minutes, once both versions have at least 50 visitors. When it's 99% sure version B is worse than A, the test stops.
- Everyone sees the current version (A) again.
- The check page says Stopped early and why. In lists and on the Timeline, it shows as Stopped · it hurt.
- You get an alert by email, and in Slack if it's connected (more). It comes with the Results emails.
- No lift is shown. A test stopped early makes the effect look bigger than it is.
- It only stops for harm. A test that's winning still waits for its full verdict.
Why checking often is safe here. Looking at an ordinary test again and again brings false alarms: sooner or later, luck dips below the line. Needlr's harm check uses a bound made to be read at every update. It stays 99% sure however many times it looks. In Needlr's simulation of healthy tests read every day for 30 days, it stopped fewer than 2% of them.
Guardrail KPIs. An agent can name up to two more KPIs that must not drop, like checkout started. The same check watches them. If B clearly hurts one, the test stops too, and the page says it was a guardrail KPI. A guardrail never decides the verdict, but a win that hurt a guardrail isn't recommended as a keep.
Net 2: no false wins
Flip a fair coin 10 times and you'll often see a run of 4 heads. Nothing is wrong with the coin. Run 50 tests and a few will “win” by luck alone.
So Needlr looks at all the agents' split tests in a project as one line, in the order their verdicts came. Each new win must clear its own bar. The bar depends on the tests before it.
- The bar is much stricter than one test's usual 95%. The first agent win in a project needs very strong evidence.
- Each confirmed win loosens the bar a little for the tests after it.
- Over time, the share of false wins stays at 5% or less, on average.
A win that doesn't clear its bar still says Improved, with a Not confirmed label and this note: test it again before you keep it. The agent is told to iterate rather than keep it. It doesn't count as a proven win on the agents' scorecard.
What stays the same
- Agents' split tests are judged at 95% confidence and wait at least 7 full days, like any split test (why).
- Before vs after impact checks an agent asks for are judged like anyone's: likely, not proof.
- Split tests people start don't get these two nets.
Common mistakes
- Keeping a Not confirmed win. Run it again first. A second win makes luck much less likely.
- Rerunning a test that stopped for harm. It hurt with 99% certainty. Change the idea, don't just run it again.
- Reading a stopped test's early numbers as the effect. An early stop makes it look bigger than it is.
- Harm bound
- The line Needlr checks at every update. Below it, B is worse with 99% certainty, and the test stops.
- Guardrail KPI
- A KPI that must not drop. It can stop a test, but never decides the verdict.
- False win
- A test that says improved when the change didn't really help.
- Not confirmed
- A win that didn't clear its bar among the project's agent tests. It could be luck.
What the video says
Agents run test after test, and nobody watches each one. That changes the math. Needlr adds two safety nets to every split test an agent runs.
First, harm. A bad change costs you every day it runs. So Needlr checks each agent's split test at every update, and stops it as soon as version B is clearly hurting.
Checking often usually breeds false alarms. Needlr's harm bound is built for constant looks: it stops only when it's 99% sure B is worse, however often it looked.
The same net guards up to two guardrail KPIs the agent names. When it stops a test, everyone sees the current version again, and you get an email or Slack alert.
Second, luck. Run enough tests and some wins are flukes. Needlr judges each agent win against every agent test before it, keeping the share of false wins at 5% or less.
A win that doesn't clear its bar is marked Not confirmed, and the agent is told to try again rather than keep it. Fewer false winners shipped, more trust in every keep.