Split tests

Split test in code

Your developer wraps the new version in one line of code. Needlr decides who gets it, and counts your KPI.

In Needlr: New check, step 1 (Split test (developer))

0:00 / 1:06

Some changes live deep in your product: a new checkout flow, a smarter search, a different onboarding. Split test developer is the impact check for changes your team builds in code.

A split test in code is for changes your developers build: a new checkout, a smarter search, a different onboarding. Your developer adds one line of code, and Needlr decides which visitors get the new version.

Think of a light switch

Your developer builds the new feature and puts a light switch in front of it. On: visitors see the new version. Off: they see the current one.

Needlr holds the switch. For each visitor it flips a coin, and remembers the answer. So the same visitor gets the same version every time they come back.

When to use it

Good for

A change that needs new code: a new feature, a new flow, new logic, new prices.
Changes inside your app, after people log in.
Anything you can't change by clicking on the page in the visual editor.

Use something else when

Words, buttons, colors or a new page you can change yourself: a split test with the visual editor needs no developer.
A change already live for everyone: use Before vs after.
Counting how many people use a new feature: use Adoption.

Before you start

Needlr must already run on your site, through Google Tag Manager. Starting the impact check needs Tag Manager connected, and a code from your authenticator app.

How to set it up

1On your project page, press + New impact check. Under What are you shipping?, give it a short name, like New checkout.
2Pick Split test (developer). Needlr shows the code, with your impact check's name in it: “New checkout” becomes new_checkout.
3Press Copy and send it to your developer. They put the new version in the first part and the current version after else.
4Which needle should it move? Pick the KPI, like a purchase or a signup.
5How should we test it? Choose how many visitors get the new version and how sure you want to be (more).
6Press Start measuring when the code is live. Not yet? Press Save as draft. The check page then shows the code under What version B gets, and a Start button for later.

your-app / checkout.tsx

// checkout.tsx

if (await impact.feature("one_click_checkout")) {

showOneClickCheckout();

} else {

showCurrentCheckout();

}

One line from Needlr

The line from Needlr goes around the new version and the current one.

The line can go live before you start. Until the impact check runs, it answers “current version” for everyone, so nobody sees the new one by accident.

Who gets which version

Random and anonymous. The first time someone visits, Needlr gives them a random ID in their browser. The version comes from that ID (how).
Sticky. The same visitor gets the same version on every visit.
Never slow. If Needlr can't answer within 0.8 seconds, the line answers “current version”, so the page never waits.
Your share. If you chose 50%, about half the visitors get the new version.

How counting works

A visitor is counted when the line runs for them, on their very first visit too. Only people who reach that part of your product are in the impact check, in both groups alike.
Then Needlr counts your KPI in each group, from that page on, and compares the share of people who did it.
Put the line where the change happens. If it runs on a page people never see, they're counted but never see the change.

When there's a verdict

The verdict comes when the planned number of visitors is in and at least a full week has passed (why). The check page then says Verdict ready. The split keeps running until you act:

Improved

Press Ship version B. From then on, the line answers “new version” for every visitor. No deploy needed (more).

Got worse

Press Stop the test. The line answers “current version” for everyone again.

No clear effect

Stop the test, and keep whichever version is simpler for your team.

Moving the winner into your code

1Your developer deletes the old version and the line, so the new version runs on its own. Then they deploy.
2On the Shipped changes tab, press Ready for verification.
3Needlr can't check code changes by itself, so it says Needs a manual check. Check the live site, then press I checked it works without Needlr.
4Press Stop applying through Needlr.

Watch outDo it in this order. If Needlr stops applying it while the line is still in your code, the line answers “current version” again, and everyone gets the old version.

Common mistakes

Changing the name in the code. The name in the line must stay exactly as Needlr shows it, or Needlr can't find your impact check.
Deleting the line before shipping or stopping. Then nobody gets version B, and the counting stops.

Words to know

Feature flag: A switch in the code that turns a new version on or off. Needlr's line is one.
Branch: One of the two paths in the code: the new version, or the current one after else.
Deploy: Putting new code on your live site or app.

What the video says

Some changes live deep in your product: a new checkout flow, a smarter search, a different onboarding. Split test developer is the impact check for changes your team builds in code.

Needlr gives your developer one line that asks Needlr: does this visitor get the new version? If yes, show it. If not, the current one. That's the whole integration.

Needlr picks the version from an anonymous visitor ID, at random, and it sticks across visits. If Needlr can't answer within a split second, the code simply shows the current version.

A visitor is counted the moment your code asks, in both groups alike. Then Needlr counts your KPI in each group: the share who sign up, buy, or click.

With the planned sample and a full week in, you get the call. The split keeps running until you act: ship the winner, or stop, and everyone gets the current version.

Ship it, and the line answers yes for every visitor, no deploy needed. Later, your developer deletes the old branch and the line. Bold product bets, settled with evidence.