Data problem: why a check is on hold

When the counting breaks, Needlr pauses the verdict instead of guessing. Here's what to fix.

Data problem means Needlr found something wrong with the counting. It puts the impact check on hold: no verdict and no numbers, until the problem goes away or you start again.

Think of a kitchen scale

If your kitchen scale says 2 kg with nothing on it, you don't weigh the flour. You fix the scale first. Any number it shows now would be wrong.

Needlr does the same. When the counting is broken, it won't show you a result, even a good-looking one.

What you'll see

• The label above the needle says On hold. The needle is grey and the numbers are hidden.
• The sentence under it says what's wrong, and a yellow What to do box says how to fix it.
• The live panel is hidden while the impact check is on hold.
• On your project page, the impact check shows On hold.

Reason 1: the split is uneven

Say you planned half the visitors on version A and half on B. Chance makes it a little uneven, but only a little. When it's far more uneven than chance allows, something is choosing who sees which version. Experts call this a sample ratio mismatch.

The page says, for example: “The split between the versions is off (6,120 vs 4,880 visitors), so the numbers can't be trusted.” Needlr starts checking this once an impact check has 200 visitors.

Common causes: a redirect, a page cache, or a bot that only lands on one version. Or the new version fails to load for some visitors, so they are never counted.

1. 1Press Preview B at the top of the impact check's page (visual split tests). Check that the page loads and looks normal. For a developer split test, ask your developer to check that both versions show correctly.
2. 2Fix what's wrong.
3. 3Press Stop it, then start fresh in the What to do box, and confirm. The impact check stops and everyone sees the current version.
4. 4Press Run it again. It makes a fresh draft with the same change and KPI. Start it (how).

Reason 2: your KPI stopped firing

Needlr watches every KPI it counts on your site. If a KPI used to happen every day and, in the last 24 hours, fell below a tenth of its normal daily count, Needlr calls it stopped (KPI health).

Usually something on your site changed: the button's words, the page's address, or the form. Owners and Editors get an email when a KPI stops, and another when it fires again.

1. 1Find what changed. Does the button or page your KPI counts still exist on your site?
2. 2Make the KPI fire again: put back what changed, or make sure your latest changes are published to Tag Manager.
3. 3Wait. Results come back by themselves once the KPI fires again.

KPIs counted by Google Analytics aren't watched this way, so they never cause this kind of hold.

Stuck, but not on hold

These look like problems, but the impact check isn't on hold:

• At this pace this check won't reach a verdict. Too few visitors reach the impact check, or the element you changed shows before Needlr loads. The box says what to do (test size).
• Unusual traffic excluded. One source sent far more visitors than normal. Needlr left them out and carries on. Nothing to fix.
• This check will never have a “before”. A before vs after impact check with no history to compare with (more).

Common mistakes

• Waiting for an uneven split to fix itself. It won't. Fix the cause and start fresh.
• Trusting the numbers from before the problem. They were collected with the same broken counting.
• Seeing a hold as a failure. It's Needlr refusing to give you a wrong answer.

Words to know

Sample ratio mismatch

The visitors aren't split the way you planned, by far more than chance allows.

KPI firing

Your KPI being counted when people do it on your site.

On hold

No verdict and no numbers, until the counting is right again.