Impact measurement
Know whether your last change actually helped
Every fix ROOKON proposes, and every release you log, gets an impact report: the same metrics in equal windows before and after, a significance test for each, and a plain verdict. Including when a change made no difference, or made things worse.
The problem
Most teams ship and hope
A change goes out, the dashboard wiggles, and someone declares victory, or nobody looks at all. Without a fair comparison you can’t tell a real improvement from a busy Tuesday, and the next decision is made on a hunch.
- Before/after comparisons with windows of different lengths
- No check whether a difference is bigger than normal day-to-day noise
- Other releases in the same week muddy the picture
- Nobody follows up once the ticket is closed
Live demo 1 of 2
Explore an impact report
Simulated: six shipped changes for a fictional SaaS (ROOKON fixes, a release and a manual note), each measured over equal windows.
- Click Go back on the release that made bounce rate worse, then watch the recovery
- Switch to a 7-day window
- Find the change flagged for overlapping another
rookon · acme-cloud · impact of changes
Simulated dataEach change: 14 days before vs. 14 days after.
12 NovLive
Conversion rate · pages ROOKON changed
4.67% → 5.85%
+1.2 pp since the first ROOKON change
Bounce rate · same pages
41.4% → 38.5%
−2.9 pp since the first ROOKON change
ROOKON changes
2 improved of 3 measured
1 no significant change · 0 worse · 1 collecting
Conversion rate · /billing · 7-day rolling
| Time | conversion rate |
|---|---|
| 14 Sept | 3.1% |
| 15 Sept | 3.1% |
| 16 Sept | 2.7% |
| 17 Sept | 2.5% |
| 18 Sept | 2.5% |
| 19 Sept | 2.4% |
| 20 Sept | 2.5% |
| 21 Sept | 2.8% |
| 22 Sept | 2.7% |
| 23 Sept | 2.7% |
| 24 Sept | 2.9% |
| 25 Sept | 2.9% |
| 26 Sept | 2.9% |
| 27 Sept | 3.0% |
| 28 Sept | 2.9% |
| 29 Sept | 3.2% |
| 30 Sept | 3.5% |
| 1 Oct | 3.4% |
| 2 Oct | 3.6% |
| 3 Oct | 3.7% |
| 4 Oct | 3.5% |
| 5 Oct | 3.6% |
| 6 Oct | 3.5% |
| 7 Oct | 3.4% |
| 8 Oct | 3.7% |
| 9 Oct | 3.3% |
| 10 Oct | 3.1% |
| 11 Oct | 3.5% |
| 12 Oct | 3.8% |
| 13 Oct | 3.9% |
| 14 Oct | 3.7% |
| 15 Oct | 3.2% |
| 16 Oct | 3.6% |
| 17 Oct | 3.9% |
| 18 Oct | 3.9% |
| 19 Oct | 3.7% |
| 20 Oct | 3.7% |
| 21 Oct | 3.8% |
| 22 Oct | 4.4% |
| 23 Oct | 4.6% |
| 24 Oct | 4.5% |
| 25 Oct | 4.3% |
| 26 Oct | 4.3% |
| 27 Oct | 4.3% |
| 28 Oct | 4.4% |
| 29 Oct | 3.9% |
| 30 Oct | 3.7% |
| 31 Oct | 3.6% |
| 1 Nov | 3.7% |
| 2 Nov | 3.8% |
| 3 Nov | 3.7% |
| 4 Nov | 3.7% |
| 5 Nov | 3.8% |
| 6 Nov | 3.7% |
| 7 Nov | 3.7% |
| 8 Nov | 3.6% |
| 9 Nov | 3.7% |
| 10 Nov | 3.6% |
| 11 Nov | 3.7% |
| 12 Nov | 4.1% |
Shaded: baseline and after windows of the selected change.
It was better before
“Release v2.15.0: new onboarding checklist”
Bounce rate +2.6 pp (32.9% → 35.5%)
p < 0.001 (Holm-adjusted) · 14 days before vs. after
Either way, ROOKON won’t propose this exact change again. A different approach to the same page is still allowed.
Overlaps with “Release v2.15.0: new onboarding checklist”, so part of the difference may come from that change.
Observed changes between equal windows before and after each change, tested for significance (two-proportion z-test for rates, Welch’s test for durations, Holm-adjusted, p < 0.05). This shows association, not proof of cause: seasonality, campaigns and other releases can contribute, so overlapping changes are flagged.
How ROOKON helps
What you get
Automatic for every ROOKON change
When a proposed pull request is merged, ROOKON records the moment and starts measuring. You can log your own releases and experiments too.
Fair windows
The same number of days before and after, so weekdays and weekends balance out. Choose 7, 14 or 28 days.
Significance, not vibes
Rates are tested with a two-proportion z-test and durations with Welch’s test, adjusted for testing several metrics at once. Without p < 0.05, there’s no verdict.
No early peeking
A change stays “collecting” until its window is complete, so early noise is never reported as a result.
Honest about confounders
Other changes inside the same window are flagged, because part of the difference may come from them.
An honest scorecard
See how many changes improved things, made no difference or made them worse. When it was better before, ROOKON offers to go back and won’t propose that exact change again.
Live demo 2 of 2
Where measurement starts: the merge
The last step of a simulated ROOKON change: you merge and deploy, and ROOKON starts watching the same signals that triggered the fix.
- Step back to see the change itself
github.com/acme/acme-cloud · pull/128
Simulated dataYou merge. You deploy. ROOKON verifies.
You merge in GitHub (or tap Merge in ROOKON) and ship with your existing pipeline. ROOKON never merges on its own or touches production. After your release, it keeps watching the same signals to confirm the fix worked.
After release v2.14.2 Live
1,240 sessions on /billing · 41 upgrades · 0 TypeErrors
- TypeError in PlanSelector: no new events
- Rage clicks on “Upgrade to Pro”: back to baseline
- Issue closed automatically, with a summary
How a change gets measured
- 01
A change ships
A ROOKON pull request is merged, or you log a release or an experiment with its timestamp.
- 02
The baseline is fixed
The window before the change becomes the baseline for each metric on the affected pages.
- 03
Data is collected
The after-window fills with the same metrics. No verdict until it is complete.
- 04
Verdict
Each metric gets before, after, change, an adjusted p-value and a verdict: improved, worse, no significant change, or not enough data. If it got worse, ROOKON suggests going back.
FAQ
Common questions
Does an improvement prove the change caused it?
No, and ROOKON never says it does. A before/after comparison shows an association. Seasonality, campaigns and other releases can contribute, which is why overlapping changes are flagged. For causal proof you need a controlled experiment such as an A/B test.
Which metrics are measured?
Conversion rate for each goal, bounce rate, average session duration and traffic volume, for the pages the change touched or for the whole site.
What does “not enough data” mean?
Some pages don’t get enough visits for a reliable answer within the window. Rather than guess, ROOKON shows the numbers and withholds a verdict until there is enough data or a longer window is chosen.
Will ROOKON show changes that made things worse?
Yes. A scorecard that only shows wins isn’t worth trusting. Worse results are shown exactly like improvements, including for changes ROOKON proposed.
What happens if a change makes things worse?
Once a change’s 14-day after-window is complete, ROOKON checks it. If conversion or bounce rate got significantly worse and no key metric improved, you see “It was better before” on the Impact page, with the numbers, and get an alert if you’ve set up email or webhook alerts. Choose Go back and ROOKON logs a revert marker, gives you a short checklist and measures the recovery like any other change, or keep the change and note why. Either way ROOKON remembers it and won’t propose that exact change again. A different approach to the same page is still allowed, with a warning. Automatic reverts of ROOKON’s pull requests and Shopify edits arrive with those integrations; until then you undo the change yourself and confirm it.
Can I measure releases ROOKON didn’t propose?
Yes. Log a release or experiment with a timestamp in the dashboard or through the API, and it gets the same report.
Ship changes. Know what they did.
Sign up and get an impact report for every change you make.