Before vs after
Measure a change that's already live for everyone, by comparing the weeks before it with the weeks after.
You shipped to everyone last Tuesday. No split group, no control. Did it move the needle? A before versus after impact check compares your KPI in the weeks before the release with the weeks after.
Before vs after measures a change that's already live for everyone. Needlr compares your KPI in the weeks before the release with the weeks after it.
Think of a plant
You start giving your plant a new fertilizer. A month later it's taller. Was it the fertilizer? Probably.
But it was also spring, with more sun. You can't be sure, because there was no second plant without the fertilizer to compare with.
Before vs after is like that: a strong hint, not proof. A split test is the version with two plants.
When to use it
- The change is already live for everyone.
- You can't split your visitors, for example a new price or a new app version.
- A release from the past that nobody measured (how far back).
- The change isn't live yet: a split test gives you proof.
- Another big change went live within two weeks of it: the two get mixed up.
- You want to know how many people use a new feature: use Adoption.
How to start one
- 1On your project page, press + New impact check. Under What did you ship?, give it a short name and pick Before vs after.
- 2Set the Release date: the day it went live for everyone. It can be in the past, today, or up to 90 days ahead.
- 3Which needle should it move? Pick the KPI. It needs numbers from before the release.
- 4How should we test it? Choose How sure do you want to be? and read the What happens box. Needlr warns you here if the KPI has no history from before the release.
- 5Press Start measuring.
The release date decides what happens:
- In the past: Needlr compares up to 4 weeks before that date with up to 4 weeks after it.
- Today: the 4 weeks before today with the 4 weeks to come.
- In the future: Needlr compares the 4 weeks before that day with the 4 weeks after. Useful when your KPI has no history yet: pick a day at least 4 weeks away, so the whole “before” is counted.
How it compares
Needlr takes two windows of the same length, right before and right after the release day. Each is as long as the time since the release, up to 28 days.
In each window it counts the share of visitors who did your KPI. Then it works out the difference, and a range the real difference is likely in, just like a split test.
The verdict comes when the window after the release has 28 days. For a release more than 4 weeks ago, that's right after you start. Then the impact check stops by itself.
With a Google Analytics KPI, Needlr reads Google Analytics' daily numbers instead, and nothing is published to your site.
What the result means
The KPI went up after the release. Likely thanks to it, but other things in the same weeks can play a part. Keep it.
The KPI went down after the release. If nothing else changed in those weeks, this release is the likely cause. Think about rolling it back.
The release didn't visibly move the KPI. Keep it if it's simpler for your users or your team.
Your KPI stopped firing, so the impact check is on hold until it fires again (more).
Every result says Before vs after · likely, not proof, so nobody mistakes it for a split test.
The warnings it shows
- Careful: this KPI was already moving before the release. Needlr compares the two windows before the release. If the KPI was already climbing or falling, part of the change may be that trend, not your release. It's a warning: it doesn't change the verdict.
- Careful: another release is too close. Another release went live within two weeks of this one. The comparison can't tell the two apart.
- This check will never have a “before”. Needlr started counting this KPI after the release. Use a Google Analytics event instead, or measure your next change with a split test.
- Older than 90 days: counted from daily summaries. For old releases, the numbers come from Needlr's daily history. They're accurate to about 1%, so the verdict is a little more careful.
Common mistakes
- Two releases in the same week. You'll get one answer for both. Space your releases out, or use split tests.
- A release in the middle of a big sale or campaign. The campaign moves the KPI too. Keep it in mind when you read the result.
- A brand new KPI for an old release. There's nothing to compare with. Set up your main KPIs early (why).
What the video says
You shipped to everyone last Tuesday. No split group, no control. Did it move the needle? A before versus after impact check compares your KPI in the weeks before the release with the weeks after.
Needlr takes two windows of the same length, up to four weeks each, right before and right after the release date, and compares the share of visitors who hit your KPI.
That gives you a strong hint, not proof. A campaign, a holiday or another release in the same weeks can move the KPI too. So these verdicts are labeled likely, not proof.
Needlr also checks the trend: it compares the two windows before the release. If the KPI was already climbing, you get a warning that part of the lift may be trend.
You'll also be warned when another release went live within two weeks, and when the KPI has no history from before the release, so there's nothing fair to compare.
Use before versus after for what's already out. When you need proof, test the next change with a split test, before everyone sees it.