Too slow to finish

When a split test can't reach its verdict at the pace visitors arrive, Needlr says so. Why it happens, and how to fix it.

Some split tests can't finish: too few visitors reach them to ever collect the planned number. Needlr tells you on the check page, so you don't wait for an answer that won't come.

Think of filling a bathtub

You need 200 liters, but the tap only drips. You could wait a year. Or you could use a smaller tub, or open the tap wider.

The tub is the number of visitors your impact check needs. The tap is how many visitors reach it each day. The fixes below make the tub smaller or the tap wider.

How Needlr tells you

Needlr keeps track of how many visitors reach each version each day. If, at that pace, the rest of the planned visitors would take more than about 2 months, the check page changes. When one version gets a smaller share of visitors (say 10%), that version sets the pace:

• Under the needle: At this pace this check won't reach a verdict. See below for why and what to do.
• A yellow box, At this pace this check won't reach a verdict, with your numbers. For example: “Only about 20 visitors a day reach each version, and it needs 13,914 per version.”
• The timeline says Too slow to finish at this pace, and there's no verdict date.

This is for split tests. A Before vs after check always takes 28 days after the release, so it can't be too slow this way.

A visual editor test can get the same box for another reason: the thing you changed is on screen before Needlr loads, so most visitors are left out. The box then says what to do: set the check up again choosing Change it anyway, or add the no-flicker line to your site.

Why it happens

• The page gets few visitors.
• The KPI is rare, like a purchase three steps after the page you changed.
• You're looking for a small improvement. At +10%, a test needs about four times the visitors it needs at +20%.
• The split isn't half and half, so one group fills up slowly.
• The KPI's real rate turned out lower than planned, so Needlr re-planned for more visitors.

How to fix it

A running check's settings can't be changed. Stop it, then set up a new impact check that can finish.

1. 1On the check page, press Stop (what stopping does).
2. 2On the project page, press + New impact check.
3. 3Change one or more of the things below. In step 3, the What happens box tells you how long it will take, for example First answer in about 6 weeks. Aim for weeks, not months.
4. 4Press Start measuring.

What to change

• A bolder change. In Advanced settings, pick a bigger Smallest improvement worth knowing. On a KPI at 3%, 95% sure: +20% needs 13,914 visitors per version, +50% needs 2,518. Then test an idea big enough to move it that much.
• Half and half. Set How many visitors see the new version? to 50%. That's the fastest. More or less than half is slower, because the smaller group sets the pace.
• Less sure, for low-risk changes only. At 80%, the same test needs about 40% fewer visitors (7,991 instead of 13,914). But about 1 answer in 5 could be luck, so don't do this for prices.
• A KPI closer to the change. A click on the page you changed happens far more often than a purchase three steps later. With 5,000 visitors a week and +20%: a KPI at 8% answers in about 2 weeks, one at 0.8% in about 5 months (pick a button).
• A busier page. More visitors a day fill the plan faster.

Common mistakes

• Leaving it running just in case. It won't get there. It also takes up one of the 100 places for checks measuring at once in the project (plans).
• Lowering how sure you need to be for a price change to make it fit. Make the change bolder instead.
• Showing the new version to most visitors to speed it up. It slows it down. Half and half is the fastest (why).