Which check should I use?

Four kinds of impact check, one question: is the change already live for everyone?

Needlr has four kinds of impact check. One question picks the right one: is the change already live for everyone?

Think of a new cookie recipe

You baked a new kind of cookie for your class. How do you find out if it's better?

Before you hand them out: give half the class the old cookie and half the new one, by coin flip. Count who asks for seconds. That's a split test. It's the fair way, because both halves ate on the same day.

Already handed them out to everyone: compare this week with last week. Maybe more kids asked for seconds. But maybe it was just a hungrier week. That's before vs after: a good hint, not proof.

You just want to know if kids even tried the new flavor: count how many did, and check it against your goal. That's adoption.

Decide in one question

1. 1Is the change live for everyone yet? If not, use a split test. It's the only kind that gives proof.
2. 2Can you change it on the page yourself (words, buttons, colors, hiding or moving something)? Pick Split test. You change your page in the visual editor. Or send visitors to another page instead.
3. 3Does it need code (new logic, a new checkout, a whole new flow)? Pick Split test (developer). Your developer adds one line of code that Needlr shows.
4. 4Already live for everyone, and you want to know if a KPI moved? Pick Before vs after.
5. 5Already live, and you want to know if people use the new thing? Pick Adoption.

Split test

Part of your visitors see your change, the rest see your site as it is, at the same time. Needlr compares the two groups. Because a coin flip decided who saw what, a difference is caused by your change (more).

Both split tests work the same way. The only difference is how the new version gets to visitors: you change the page in Needlr, or your developer puts it in the code.

Before vs after

For a change already released to everyone. You give the Release date. Needlr compares up to 4 weeks before that date with up to 4 weeks after it (more).

The answer is “likely, not proof”: other things happen in the same weeks, like a sale, a holiday or a new ad. Needlr warns you when the KPI was already rising or falling before the release.

It needs numbers from before the release. Either a KPI Needlr has been counting since before then, or Google Analytics. The release date can also be in the future: Needlr then collects the “before” until that day.

Adoption

For a new feature that's live for everyone. You set a Target: what share of active people should use it, and within how many days. For example 20% within 30 days. Needlr tells you if you reached it (more).

“Active people” are everyone who visits your site, unless you change it. You can limit them to people who did up to 3 things, like “logged in”.

Common mistakes

• Shipping to everyone first, then asking. You lose the chance of proof. If you can wait, run a split test first.
• Treating before vs after as proof. It's a strong hint. A big sale in the same month can fake a win.
• Using adoption to judge if a feature made money. Adoption tells you if people use it, not if it moved your KPI.