Reliability report
How often is the verdict right?
Two kinds of evidence. First, 192,000 simulated checks where we know the truth, run through the same code that decides your checks. Second, a test site browsed by real browsers, where we planted the effects and compared them with what Needlr found. How each verdict is computed is in the methodology.
1. Simulation
Each simulated check has a KPI that converts 5% of visitors and looks for an effect of +20% or more. Traffic follows a weekly pattern: fewer visitors at the weekend, and they convert a little less. Each check starts on a random weekday, splits visitors 50/50 at random and is looked at every hour. The verdict comes from the production code: the plan, the replan, the interval, the split check, the 7-day minimum and the freeze.
Each setup runs three ways: the plan made from the true rate, and plans made from a rate guessed at half or double the true one, which Needlr then replans from the real data. 4,000 checks per cell. Generated on October 9, 2026 (seed 20261008, so anyone can repeat it with node scripts/aa-simulation.mjs).
False positives: changes that do nothing (A/A)
A and B are identical. Any "improved" or "worse" is a false positive. It should happen at most as often as the confidence level allows: 5% at 95%. The last column shows what a dashboard that kept recomputing after the verdict would have shown at some point, up to twice the plan.
| Confidence | Promised | Plan from true rate | Guessed 0.5x, replanned | Guessed 2x, replanned | If recomputed after the verdict |
|---|---|---|---|---|---|
| Steady traffic: the plan takes about 2 weeks. The planned sample decides when the verdict comes. | |||||
| 95% | 5% | 4.9% | 4.9% | 5.8% | 16.3% |
| 90% | 10% | 9.8% | 9.6% | 9.6% | 28.7% |
| 80% | 20% | 20.6% | 20.1% | 19.8% | 49.6% |
| 70% | 30% | 30.3% | 30.4% | 29.6% | 65.6% |
| Busy site: the plan takes about 3 days. The full-week minimum decides. The verdict then has more data than planned, so power is higher. | |||||
| 95% | 5% | 4.8% | 5.7% | 4.8% | - |
| 90% | 10% | 10.2% | 9.6% | 10.2% | - |
| 80% | 20% | 19.7% | 20.6% | 19.9% | - |
| 70% | 30% | 29.8% | 29.5% | 30.8% | - |
Standard error at 4,000 checks: about 0.3 points at 95%, 0.7 at 70%. A value in amber would be more than 3 standard errors above the promise.
Power: finding a real effect of the planned size
B really is +20% better. The plan is built to find that 80% of the time. The table shows the share of checks whose verdict was "improved".
| Confidence | Planned | Plan from true rate | Guessed 0.5x, replanned | Guessed 2x, replanned | Days to verdict |
|---|---|---|---|---|---|
| Steady traffic: the plan takes about 2 weeks. The planned sample decides when the verdict comes. | |||||
| 95% | 80% | 80.5% | 80.1% | 79.8% | 14 |
| 90% | 80% | 79.9% | 80.2% | 79.6% | 14 |
| 80% | 80% | 79.4% | 80.3% | 79.5% | 14 |
| 70% | 80% | 79.9% | 80.8% | 80.1% | 14 |
| Busy site: the plan takes about 3 days. The full-week minimum decides. The verdict then has more data than planned, so power is higher. | |||||
| 95% | 80% | 99.0% | 99.1% | 99.2% | 7 |
| 90% | 80% | 98.2% | 98.7% | 98.5% | 7 |
| 80% | 80% | 97.4% | 97.9% | 97.3% | 7 |
| 70% | 80% | 96.2% | 97.0% | 96.8% | 7 |
Standard error at 4,000 checks: about 0.6 points. Days to verdict: the median, for the plan from the true rate.
What this shows
- The promise holds. At every level, with or without a replan, the share of do-nothing changes called a winner or a loser stays at the level's rate: about 1 in 20 at 95%, 3 in 10 at 70%.
- Freezing the verdict matters. Recomputed after the verdict, 16% of do-nothing changes would have shown a result at some point at 95%, about 3 times the promise. That's why a verdict never changes.
- Planned power holds. With a plan from the true rate, real effects of the planned size are found 79.4% to 80.5% of the time, against 80% planned.
- Replanning keeps the planned power. When the plan had to be replanned from a wrong guess, power is 79.5% to 80.8%. The replan uses A's rate estimated from both versions, not A's own rate, so a check doesn't stop sooner when A happens to run high. False positives are not affected.
- A full week on a busy site gives more certainty, not less. When the sample is in after 3 days, the verdict waits for day 7 and has more data than planned.
A smaller version of this simulation runs on every code change and fails the build if false positives rise more than 3 standard errors above the promise, or power falls more than 3 below the plan.
2. Impact Lab: a known truth, real browsers
Impact Lab is a public test site for a fictional SaaS company. Real Chromium browsers visit it, and each visitor follows a truth we planted: for example, 33% fewer clicks on a new button text. Needlr measures the site like any customer's site, through Google Tag Manager. Measured on October 8, 2026.
| Check | Planted truth | Needlr measured (likely range) | Needlr's verdict | Result |
|---|---|---|---|---|
| Footer button text | −33% clicks | −35.9%−43.1% to −27.8% | Got worse | Matches the truth |
| Short signup (built in code) | +22% signups | +23.3%+18.0% to +28.7% | Improved | Matches the truth |
| Pricing page redirect | +37.5% trial clicks | +36.7%+20.5% to +55.0% | Too early, leaning up (96% of plan) | Not decided yet |
| Demo button text, at 80% confidence | +30% clicks | +32.5%+15.3% to +52.3% | Improved | Matches the truth |
| Docs button text, at 90% confidence | −25% clicks | −28.0%−39.9% to −13.7% | Too early, leaning down (70% of plan) | Not decided yet |
| Features button colour, A/A at 95% | No difference | −8.8%−20.0% to +4.1% | No clear effect | Matches the truth |
| Updates button colour, A/A at 70% | No difference | +43.5% | Improved | False positive (expected rate) |
| Broken split canary (30% of B's exposures dropped) | A broken split | Detected | Data problem, no verdict | Matches the truth |
| ROI calculator adoption | About 8% use it | 8.9%8.6% to 9.3% | On track for the 6% target | Not decided yet |
The false positive, explained
The updates button colour is an A/A check at 70% confidence: nothing changed. It was called "improved" (+43.5%). At 70%, about 3 in 10 such checks get a verdict anyway. That's the price of a fast answer. The measurement was right: Needlr's click counts (85 in A, 127 in B) matched the lab's own log. The traffic itself differed by chance.
Counting accuracy
On 4 checks, Needlr's click counts matched the test site's ground-truth log within 2 clicks. The broken split canary, where 30% of B's exposures never arrive, was caught as a data problem and given no verdict.
These split tests reached their planned samples in 1 to 2 days and were decided before the 7-day minimum existed. Verdicts never change, so they stay as decided. Checks still running now wait for a full week.
Still pending
- Before vs after for a release on October 15, measured three ways: on a Needlr KPI and on two Google Analytics KPIs. The verdicts come after 28 days, in November.
- The trend warning: a "release" with no real change, on a KPI that was already climbing. The warning should show in mid-October, about 5 days after the release.
- The ROI calculator adoption verdict, at the end of its window. The pricing redirect and the docs button verdicts, once their plan is in and they have run a full week.
We'll add each result here when it's in, whichever way it goes.