How often verdicts are right

We ran 192,000 simulated impact checks where the truth was known, and a test site with planted effects. Verdicts were wrong about as often as promised, and no more.

Needlr makes a promise: at the standard setting, a verdict is wrong only about 1 time in 20. We checked that promise in two ways. All the tables are in the reliability report.

Think of a kitchen scale

How do you know a kitchen scale is right? You put a weight on it that you already know, say 1 kilo. You look at what it shows. You do it many times.

If it shows 1 kilo almost every time, you trust it with your flour. We did the same with Needlr: we made impact checks where we already knew the answer, and counted how often Needlr got it right.

Test 1: a computer simulation

A computer made up 192,000 split tests. In half of them, the new version was exactly as good as the old one. In the other half, it was really 20% better. Each one went through the same code that decides your impact checks.

• Visitors came like on a real site: fewer at the weekend.
• Each test started on a random weekday and was looked at every hour.
• Some plans started from the right guess about the KPI. Others started from a guess half or double the truth, and Needlr had to fix the plan from the real data.

What it found

• Changes that do nothing. At 95% confidence, about 1 in 20 of them gets called a winner or a loser anyway. That's the price of any test. Needlr got 4.9%, and stayed within the promise in all 24 settings we tried.
• Changes that really work. Each split test is planned to find a real 20% lift 8 times out of 10. It found about 80% of them, even when the first guess about the KPI was wrong.
• Busy sites do even better. When the planned visitors arrive in 3 days, the verdict still waits for a full week. With those extra visitors, real lifts were found 96% to 99% of the time.
• Freezing the verdict matters. If Needlr kept recomputing after the verdict, about 16% of do-nothing changes would show a result at some point. That's why a verdict never changes (why).

Faster answers, more mistakes

On step 3 of a new impact check you can pick 90%, 80% or 70% instead of 95%, to get an answer sooner. The simulation shows the price is exactly what Needlr says: about 1 wrong call in 10 at 90%, 1 in 5 at 80%, and 3 in 10 at 70%.

Test 2: a real test site

We also built a test website for a made-up company. Real browsers visit it, and each one follows a truth we planted. Needlr measures it like any customer's site, through Tag Manager.

• A new footer button text, planted to get 33% fewer clicks. Needlr measured 35.9% fewer and said Got worse.
• A shorter signup form, planted to get 22% more signups. Needlr measured 23.3% more and said Improved.
• A button color that changed nothing, at 70% confidence. Needlr said Improved. At 70%, that happens about 3 times in 10. It's the price of a fast answer.
• A split broken on purpose. Needlr caught it as a Data problem and gave no verdict.
• Counting. On 4 impact checks, Needlr's click counts matched the site's own log within 2 clicks.

It keeps being checked

A smaller run of the simulation runs every time Needlr's code changes. If Needlr ever got wrong more often than promised, or found real lifts less often, that check would fail.

Common mistakes

• Reading 95% as “95% better”. It's how sure the verdict is, not how big the change is.
• Expecting a winner every time. No clear effect is a real answer: the change didn't matter much.
• Treating a 70% Improved like a 95% one. It's a good hint, not proof. Wrong 3 times in 10.

Words to know

False positive

A winner or loser call for a change that really did nothing.

Power

How often a real lift of the planned size gets found. Needlr plans for 80%.

Simulation

Many made-up impact checks run by a computer, where we know the true answer.