Why 7 days, and no peeking
Why a split test waits for its planned visitors and a full week before it calls a winner.
Day two, and the new version is up 30%. It's tempting to call it and ship. Hold on: on day two, that number is mostly noise.
A split test needs time. Needlr waits for the planned number of visitors and at least 7 full days before it gives a verdict. Here's why, and what looking too early costs.
Think of a basketball player
A new player takes two shots and makes both. Are they a star? Two misses: are they bad? You can't tell yet.
After 200 shots, you know how good they are. The first few days of an impact check are those first two shots.
Reason 1: early numbers jump around
On day 2, a handful of extra sign-ups can make the new version look 30% better. The next day it can look worse. That's noise, not a result.
As more visitors come in, the swings get smaller, and the real difference shows. On the check page, the band around the line gets narrower day by day.
Reason 2: the week has a rhythm
People who visit on Monday aren't the same as people who visit on Saturday. Workdays bring buyers. Weekends bring browsers. An impact check from three busy weekdays describes those days, not your site.
So a split test never gets a verdict before 7 full days since it started, even on a busy site that has enough visitors in two days.
Reason 3: the plan
Before a split test starts, Needlr works out how many visitors each version needs (how). With that many, a real improvement of the size you chose is found 8 times out of 10. Before that, the answer isn't ready.
What peeking costs
Peeking means checking the numbers every day and stopping the first time they look good. Do that, and you'll find winners that aren't real.
Needlr ran the numbers on changes that do nothing at all. If the verdict kept updating as new visitors came in, about 16 in 100 of those changes got called Improved or Got worse. With the verdict decided once, at the plan, it's 5 in 100 at 95% (the full numbers).
What you see while you wait
- Collecting data: fewer than 100 visitors in a version, or fewer than 10 conversions in total. No numbers yet.
- Too early to call: numbers are coming in, but the plan or the week isn't complete. You see a small grey Leaning up or Leaning down. A lean is not a result.
- Needs a full week of data: the plan is in, and the page says the earliest day the verdict can come.
- A progress bar with the planned visitors per version, and about how many days are left.
- Add the verdict date to my calendar, so you come back on the right day.
When the verdict comes
It's decided once, at the planned sample and after a full week. Then it's frozen: later visitors don't change it. The check page says Decided on that day. You get an email too.
Your guess for today's KPI rate was far off? Needlr redoes the plan from the real rate after a few days. That's not peeking: it only changes how many visitors are needed, never the verdict.
What if I stop early?
You can stop an impact check at any time (how). A split test stopped before its verdict gets none. The page shows which way it was leaning and says that's not enough to act on. Press Run it again to start a fresh copy.
Common mistakes
- Calling it on a lucky day. Day 2 says the new version wins? Wait. It often fades.
- Reading the lean as a result. It's grey on purpose.
- Changing the impact check while it runs. It mixes two impact checks into one. Stop it and start a new one.
- Stopping right before the end. An impact check stopped before its verdict gets no verdict at all.
Words to know
- Peeking
- Looking at the results again and again, and stopping as soon as they look good.
- Noise
- Ups and downs that happen by chance, not because of your change.
- Planned sample
- How many visitors each version needs before a verdict.
- Weekly cycle
- The way your visitors change from weekdays to the weekend.
What the video says
Day two, and the new version is up 30%. It's tempting to call it and ship. Hold on: on day two, that number is mostly noise.
Early numbers swing wildly. With a few dozen conversions, one good afternoon can look like a big lift. As data builds up, the swings shrink and the real effect shows.
Your visitors change through the week. Monday's buyers aren't Saturday's browsers. So a split test never gets a verdict before seven full days, even when the sample fills early.
Before it starts, Needlr works out how many visitors each version needs. Until that sample is in, the impact check says too early to call, and shows only which way it leans.
The cost: if the verdict kept updating, about 16% of changes that do nothing would get called a win or a loss. Wait for the plan, and it's 5%.
So wait for the verdict. It's decided once, at the planned sample and after a full week, and then it stays put. That's an answer you can take to the board.