The new check in 3 steps

Three questions on one screen, then Start measuring. Every option, and what happens after.

A new impact check is three questions on one screen, then one button. It takes about a minute.

Think of a science fair project

Every good science project answers three things. What am I changing? What will I measure? How do I make the test fair? A new impact check asks you exactly those three, in that order.

Before you start

• Your website is added on the project's Settings tab.
• Google Tag Manager is connected (how). Without it, Needlr can't count the KPI, and the Start measuring button stays off.
• To change your page in the visual editor or pick a button on your site, Needlr must already be on your site. That happens with your first Publish to GTM.
• You're an Owner or Editor of the account. Viewers can look but not create impact checks (roles).

Open your project and press + New impact check. (In a project with no impact checks yet, the button is called Create your first impact check.)

Step 1: What are you shipping?

Type a short name in Give it a short name, like “New pricing button”. You can add a Description too: what changed and why. It's optional, but your team will thank you later.

Then pick one of four cards. Not sure which? Which check should I use?

• Split test: change your site visually; part of your visitors see it. Press Open my site and make the change. Your site opens in a new tab with Needlr's editor. Click what you want to change, change it, then press Use these changes. Only you see the editor. Visitors see nothing new until you start measuring.
• Before vs after: already released to everyone. Pick the Release date. A date in the past compares up to 4 weeks before it with up to 4 weeks after it. Today compares the 4 weeks before with the 4 weeks to come. A date in the future compares the 4 weeks before it with the 4 weeks after. Needlr needs 4 weeks of the KPI before the release, so a new KPI needs a date at least 4 weeks away.
• Adoption: released to everyone; how many people use it. Pick the Release date and a Target, like 20% of active people within 30 days. The target can be 1% to 95%, within 7 to 60 days.
• Split test (developer): one line of code; part of your visitors get it. Needlr shows the code with a Copy button. Your developer puts the new version inside it.

A few more choices may show up under Split test:

• Edit again: opens the editor again to change your change.
• Show the change: Only on the page you made it on (the default), or On every page where the same element appears.
• This element is at the top of the page: visitors see the top of the page before Needlr loads. Change it anyway (recommended) may show a short fade, about 0.2 seconds. Never let anyone see it change leaves most visitors out, and the impact check may never finish. Or press Add one line to your site so nobody sees anything change.
• Or send visitors to another page: instead of editing, send people who open one page (is exactly, starts with or contains an address, like /pricing) to another page (like /pricing-v2).

And under Adoption, the Active people line. By default they are everyone who visits your site. Press Change to count only people who did All or Any of up to 3 things, like logging in. You can also choose When (any time, or all in the same visit) and Order (any order, or one after the other).

Step 2: Which needle should it move?

Pick the KPI: what you want more people to do. Click one you already have, or press + New KPI and build one: Click a button, Reach a page, Submit a form, or an event your site sends. Every option is in Create a KPI.

In an Adoption impact check this step is called What counts as using it?: the action that shows someone used the new thing.

Step 3: How should we test it?

• How many visitors see the new version? From 1% to 99%. Half and half (the default) gives the fastest answer. Fewer means less risk, but a longer wait. (Split tests only.)
• How sure do you want to be? From 70% to 95%. 95% is the default and the safest: use it for prices and big changes. Lower answers sooner, but is wrong more often. At 70%, about 3 answers in 10 could be luck.
• What happens: one plain sentence about your impact check, and when the first answer should come, like “First answer in about 3 weeks.”
• Advanced settings (split tests only): Today's KPI rate, which Needlr fills in from your site when it can, and Smallest improvement worth knowing: +10%, +20% (the default), +30% or +50%. A smaller improvement needs more visitors per group (more).
• Preview B on my site (visual split tests): see the new version on your site before anyone else. This saves the impact check as a draft. Nothing is measured until you start.

Yellow notes may appear here too. They tell you what's missing, like a release date, or a KPI with no history from before the release. Read them before you start.

Press Start measuring

1. 1Press Start measuring. It saves the impact check, publishes the KPI to Google Tag Manager and starts measuring.
2. 2Enter the code from your authenticator app. One code covers the next 20 minutes (more). A before vs after impact check counted by Google Analytics needs no code: nothing changes on your site.
3. 3You land on the check page. It shows Collecting data, then Too early to call, and then the verdict (how to read it).
4. 4You get an email when there's a result.

Drafts

Not ready to start? Press Save as draft under Start measuring. Needlr keeps your answers as a draft: nothing is published or measured. Two other things save a draft too: pressing Preview B on my site, or a start that didn't work (for example, Tag Manager couldn't be reached). Then the check page says what went wrong.

A draft shows as Draft on your project page. Open it to change its KPI (Edit KPI), to Start it, or to Delete it. If you leave the new impact check screen without starting or saving a draft, your answers aren't kept.

Common mistakes

• Vague names. “Test 3” means nothing in a month. Name the change: “Shorter signup form”.
• Lowering how sure you want to be, just to finish faster. Fine for a button's words, risky for prices.
• Stopping early. The first days swing a lot. Let it run until the verdict (why).