Releases close together
When two releases go live within two weeks of each other, a before vs after impact check can't tell which one moved the KPI.
You shipped a new pricing page on Monday and a shorter signup the Thursday after. The KPI jumped. Which release moved the needle? With before versus after alone, you can't tell.
If two of your releases went live within two weeks of each other, a before vs after impact check can't tell them apart. Needlr warns you, so nobody gives one release the credit for both.
Think of two cooks
Two cooks change the soup on the same day. One adds salt, the other adds pepper. Everyone says the soup is better. Was it the salt, or the pepper?
You can't know. You only tasted the soup before and after, and both changes happened in between.
Why it happens
A before vs after impact check compares your KPI in the weeks before a release with the weeks after it. If another release went live in those weeks, the “after” has both changes in it.
A split test doesn't have this problem. Both groups get the other release at the same time, so only your change is different.
When Needlr warns you
- The impact check is a Before vs after.
- Another release in the same project went live 14 days or less before or after it.
- The other release can be any impact check that isn't a draft: a before vs after, an adoption, or a split test. A split test counts from the day it started.
- Releases in your other projects don't count.
It's a warning only. The verdict, the numbers and the range stay exactly the same.
Where you'll see it
- On the check page: a yellow box says Careful: another release is too close, with the other release's name: it went live within two weeks of this one, and this comparison can't separate the two (check page).
- On the Timeline tab: a ! next to the release's name, and a line under it: Released within 2 weeks of the other release, with its date, and Before vs after can't separate them (timeline).
How to read the result
- Read it as both changes together. “Improved +12%” means the two releases, together, came with a 12% lift.
- Look at the other release's result too. If it was a split test, its result is proof of its own share.
- Don't add the two up. Two before vs after results that overlap can count the same lift twice.
How to avoid it
- Space big releases more than two weeks apart when you plan to measure them with before vs after.
- Split test one of them. A split test isn't confused by other releases.
- Release together on purpose, and measure them as one change. Name the impact check after both.
Common mistakes
- Thinking the warning changed the verdict. It didn't. It tells you how to read it.
- Giving the credit to the release you like better. Without a split test, the two can't be pulled apart.
- Forgetting small releases. Only releases you measured in Needlr are checked. A change you never measured can still sit in the same weeks.
What the video says
You shipped a new pricing page on Monday and a shorter signup the Thursday after. The KPI jumped. Which release moved the needle? With before versus after alone, you can't tell.
A before versus after impact check compares the weeks before a release with the weeks after. A second release inside those weeks lands in the after, so the two share the credit.
So Needlr flags a before versus after release when another release in the same project went live within two weeks of it, on either side. It's a warning: the verdict doesn't change.
On the check page, you'll see Careful: another release is too close, with the other release's name. Read the result as the effect of both changes together.
On the Timeline, the release gets an exclamation mark, and a line naming the release it sits next to, with its date. Nobody counts the same win twice.
The fix is planning. Space big releases more than two weeks apart, or split test one of them, so every win has a single, clear owner.