How Needlr reaches a verdict

A visitor counts from their first exposure: the first time their version was actually shown. Views that Needlr left unchanged, because the element was already on screen, are not exposures. A conversion counts once per visitor, only if it happened at or after the exposure (with 60 seconds allowed for clock differences) and, when the KPI has one, within its conversion window. Your team's own preview traffic is kept apart and never counted.

The KPI is binary per visitor (converted or not). For each version, the conversion rate gets a Wilson score interval at the check's confidence level. The interval of the difference B minus A is the Newcombe hybrid score interval (Newcombe 1998, method 10), built from the two Wilson intervals:

z      = Φ⁻¹(1 − α/2),  α = 1 − confidence
diff   = pB − pA
low    = diff − √((pB − lB)² + (uA − pA)²)
high   = diff + √((uB − pB)² + (pA − lA)²)
(lA, uA), (lB, uB): Wilson intervals of A and B at z

Verdict: improved when low > 0, worse when high < 0, otherwise no clear effect. The relative lift (pB / pA − 1) is shown with an interval from the log ratio (delta method, SE = √((1 − pB)/xB + (1 − pA)/xA)). A two-sided pooled z-test p-value is stored too, but the verdict uses the interval only. Nothing is shown as more than "collecting" before 100 visitors in the smaller version and 10 conversions in total.

Visitors per version for a two-sided two-proportion z-test, with p1 the expected KPI rate of A (from the KPI's history, or 3% when there is none), the minimum detectable effect (MDE) relative, and 80% power:

p2 = p1 × (1 + MDE),  p̄ = (p1 + p2) / 2
n  = [ z(1−α/2) × √(2 p̄ (1 − p̄)) + z(power) × √(p1 (1 − p1) + p2 (1 − p2)) ]² / (p2 − p1)²

The plan is reached when the smaller version has n visitors.

Results are recomputed every 5 minutes. The first time the state is final (improved, worse or no clear effect), that state is the verdict: its time is stamped on the check and no result is written after it, whether the check keeps running or is stopped. The sample ratio check is not re-run after the verdict either. There is one look that counts, so the false-positive rate is the nominal α. Verdicts given before a rule changed keep the rule they were decided under.

A split test (code or no-code) gets no verdict before 7 full days since it started. Its progress is the smaller of the sample progress (smaller version over the plan) and the days progress (days over 7). Until both are complete the state is "too early". A check stopped before 7 days gets no verdict. Before vs after and adoption checks keep their own windows (sections 10 and 11).

The plan starts from an assumed KPI rate. Once version A has at least 2 days, 300 visitors and 10 conversions, and the real rate is at most 0.67x or at least 1.5x the assumed one, the plan is recomputed with the formula above from the real rate (same MDE, α and power). It is recomputed at every update from the data so far. The check page says when this happened.

The real rate is A's, estimated from both versions: the pooled rate of A and B divided by (1 + MDE/2) for a 50/50 split (by 1 + MDE × B's share otherwise). If B really is better by the MDE, the pooled rate runs that much above A's. We don't use A's own rate because it moves with the difference the verdict tests: when A happened to run high, the plan shrank and the check stopped a little early, which cost about 1 to 1.5 points of power. The pooled rate doesn't, so a replanned check keeps its planned 80% power, and the false-positive rate is unaffected. See the reliability report.

A chi-square test (1 degree of freedom) of the observed split against the planned weights. With at least 200 visitors in total and p < 0.001, the check shows a data problem and gives no verdict while it lasts.

For KPIs Needlr counts itself, the baseline is the average daily count over the 7 days before the last 24 hours. If that baseline is at least 3 a day and the last 24 hours have less than 10% of it, the KPI has a measurement problem. Every check on it goes on hold (no verdict) until it fires again. A KPI with a baseline below 3 a day is never flagged. Google Analytics KPIs are not checked this way.

Each event is tagged with its source: a keyed hash of the client address that changes every UTC day (the address itself is never stored). A source is unusual when it brings at least 25 visitors and 5% of the check's visitors, or at least 6 conversions and 5% of its conversions. Its visitors are left out of the numbers, and the result says how many. The same rule (one database function) feeds the stored results and the live numbers. Separately, one source can add at most 50 new visitors to a project per UTC day.

Limits: a sender spreading traffic over many addresses or days stays under these thresholds. Google Analytics KPIs carry no source, so nothing can be left out there.

For a change released to everyone: the KPI rate is visitors who hit the KPI over visitors with a visit (distinct visitors, one visit a day), or Google Analytics' daily users and converters for a GA4 KPI. Two windows of equal length L, right before and right after the release date (00:00 UTC), where L is the days since the release, up to 28. The comparison is the same interval as section 2. The verdict comes once the after window has 28 days, with at least 100 visitors in each window and 10 conversions, and the check then stops.

Trend guard. A placebo comparison of the two windows before the release (L days each) at 95%, when both have at least 100 visitors. If its interval excludes zero, the KPI was already moving on its own, and the result warns that part of the change may be trend. Releases within two weeks of each other are flagged too.

Limits: the guard only sees a trend that already showed before the release. It warns; it doesn't change the verdict. Seasonality, campaigns and other changes inside the after window are not controlled. That's why these verdicts are "likely, not proof".

Raw events are deleted after 90 days. Before that, each complete UTC day is summed up per project: for visits and for each KPI, the number of hits, the exact number of distinct visitors that day, and a HyperLogLog sketch (16,384 registers) of those visitors, built from a keyed hash of the random visitor id. The summaries keep no visitor id and are kept 13 months. Internal traffic and the visitors of a source that was unusual that day (section 9, applied per day) are left out.

When any window of a before vs after check starts more than 88 days ago, all of its windows are counted from the summaries, never part from raw events and part from summaries. Merging the daily sketches gives a window's distinct visitors and KPI users within about 0.8% (standard error; in our simulations 95% of 7- and 28-day windows are within 2%). Windows of up to about 3,000 people are counted exactly. That error is added to the interval of section 2, so a verdict from summaries is a little more careful than one from raw events, never bolder. On big sites with small changes it says "no clear change" more often than raw events would. The check says when its numbers come from summaries.

Of the people active since the release (a visit, or a chosen "active" KPI), the share who used the feature (the KPI) at least once. A signed-in person is one person across devices. The share gets a Wilson interval at the check's confidence. One verdict, at the end of the target window: target reached when the lower bound is at or above the target, below target when the upper bound is below it, otherwise near the target. With fewer than 100 active people at the end, the check stops without a verdict.

Active people can also be the people who did all (or any) of up to 3 things, at any time or in the same visit (one person on one UTC day), in any order or one after another. A person counts as active from the day they first did them, and a use of the feature counts only from that day on.

Each check is judged on one primary KPI, chosen before it starts. Because only one KPI decides, there is no multiple-comparison problem in the verdict, and no correction is applied. A coding agent's split test can name up to two guardrail KPIs (section 14): they never make a verdict, they can only stop a test or hold back a "keep".

Agents run many tests, one after another, and nobody watches each one. Their split tests follow everything above, plus four rules. Confidence is 95%, whatever the agent asks for. No peeking: until the verdict, an agent sees progress and the expected date only, never which way the test leans. At most two of an agent's split tests run at once on the same KPI of a project.

Auto-stop on harm. At every snapshot, a one-sided always-valid bound (a Gaussian mixture confidence sequence) is checked on B − A, for the KPI and for each guardrail KPI. When its upper end is below zero, B is worse with 99% confidence however often it was looked at, and the test stops: everyone gets the current version. In our simulation of A/A tests read every day for 30 days, it stopped fewer than 2% of them. It only stops for harm; a win still waits for the planned verdict, and a stopped test shows no effect size (an early stop overstates it).

False wins across many tests. Each project's stream of agents' split tests is controlled with LORD++, an online false-discovery procedure: every new verdict gets its own significance level from the levels spent and the wins found before it, so the expected share of false wins stays at or below 5%. A win whose p-value is above its level is shown as "not confirmed", and the agent is told to try again rather than keep it. The same change that already lost isn't measured again unless it's sent as a follow-up.

Before vs after checks asked for by agents are the same as anyone's ("likely, not proof"); when other releases on the same KPI fall within two weeks, the agent is told the evidence is weak.