After the verdict

Run it again or iterate

What to do after Got worse or No clear effect, and when to run the same impact check again instead of testing a new version.

In Needlr: Check page

0:00 / 1:03

Not every impact check ends in a win. Got worse and No clear effect are answers too: one kept a loss off your site, the other stopped you betting on noise.

When an impact check ends with Got worse or No clear effect, it still did its job: you learned something. This page explains what to do next, and the difference between Run it again and starting a new impact check.

Think of baking a cake

Your cake didn't rise. You could bake the exact same recipe ten more times. One day it might rise by luck, but the recipe isn't any better.

A good baker changes something instead: more baking powder, a hotter oven. Then bakes again and sees. That's iterating. In Needlr, each new recipe is a new impact check.

After Got worse

Version B made your KPI worse. The check page says Stop it: the new version is costing you.

1If the page shows Verdict ready, the split test is still splitting traffic. Press Stop the test. Everyone gets the current version again.
2Results so far are kept on the check page.
3Ask why it hurt, and try a different idea in a new impact check (see below). Or drop the idea.

After No clear effect

There was no real difference. The page says No real difference: keep whichever version is simpler to maintain.

1Press Stop the test if it's still splitting traffic.
2Keep the simpler version. If that's B, ask your developer to build it in.
3If you still believe in the idea, make it bolder. Small tweaks often make no clear difference.

TipNeedlr can't roll back a release for you: in a before vs after impact check, the change is already out. If it says Likely hurt. Consider rolling it back., rolling back is done in your own product. An adoption check that ends Below target has a Set up a split test button to test how people find the feature (more).

Iterate: test a better version

Each new idea is a new impact check, so it gets its own clean verdict.

1On your project page, press + New impact check.
2What are you shipping? Make the new version: a bolder headline, a different offer, a new layout.
3Which needle should it move? Pick the same KPI as before from the list, so you can compare.
4How should we test it? Choose the share of visitors and how sure you need to be, then press Start measuring.

Run it again

Same change, same KPI. For a check that stopped early or broke.

New impact check

A new idea, or new settings. For everything else.

Most of the time, the next step is a new impact check.

When to use Run it again

Run it again only shows on the check page in two cases:

It stopped before its verdict. Someone pressed Stop too early. The page says what it knew so far, and that it's not enough to act on.
It's on hold because the split broke, and it has been stopped. Data collected under a broken split can't be trusted, so you start fresh (more).
Adoption checks don't have it. Start a new one instead.

It makes a fresh draft and opens it. The draft copies:

The name, the description, the change and the KPI.
How sure the verdict must be, and the smallest improvement worth knowing.
The rate it plans with, updated to the real rate if Needlr measured one.

The share of visitors who see the new version goes back to half and half. Nothing runs until you press Start on the draft. Don't need it? Press Delete.

Watch outA copy is just as slow as the original. If the first one was Too slow to finish, don't run it again: start a + New impact check on a busier page, or with a KPI that happens more often (more).

Common mistakes

Running the same change until it wins. Run enough tests and one will look like a win by luck. Change the idea, not the dice.
Leaving a finished split test running. Until you stop it or ship the winner, part of your visitors still get version B. The page says Verdict ready until you act.
Changing two things and the KPI at once. Keep the KPI the same between versions, so you know what made the difference.
Running it again after it was too slow. It will be too slow again.

What the video says

Not every impact check ends in a win. Got worse and No clear effect are answers too: one kept a loss off your site, the other stopped you betting on noise.

First, close it out. A split test with a verdict keeps splitting traffic until you act. Press Stop the test, and everyone gets the current version again.

Then iterate. Keep the same KPI, change the idea: a bolder headline, a different offer. Each new version is a new impact check, with its own clean verdict.

Run it again is for a different case: an impact check that stopped before its verdict, or a broken split. It makes a fresh draft with the same change and KPI.

The copy keeps the settings: how sure it must be, the smallest improvement worth knowing, the real rate measured so far. A test that was too slow will be just as slow.

One rule: don't rerun a finished impact check hoping for another answer. Roll the dice enough times and noise looks like a winner. Change the idea, not the dice.