Free tool
Lighthouse required-runs calculator
“We ran it five times and the score went up two points” isn't evidence — Lighthouse is noisy enough that a small before/after change can be pure run-to-run variance. This tool measures your page's real variance for you (a short PageSpeed Insights pilot, per metric), tells you how many runs a claim actually needs, writes the experiment protocol — and when you've run it, drop your before/after Lighthouse JSONs below for the statistical verdict.
n = ⌈2·(z₁₋α/₂+z_power)²·σ²/δ²⌉
- Metric
- Performance score
- Run-to-run σ
- 2.7 preset · Heričko-derived
- Change to detect (δ)
- 3 points
- Comparison
- independent runs
- α / power
- 0.05 / 0.8
That σ is an assumption, not your page. σ enters the formula squared, so the run count moves fast when it is wrong — with a short PageSpeed pilot.
Already ran it? Check a result.
Enter the change you measured and how many runs produced it — you get a plain verdict against this page's noise.
| Runs n | 95% CI half-width | Min. detectable Δ |
|---|---|---|
| 3 | ±4.32 | 6.18 |
| 5 | ±3.35 | 4.78 |
| 7 | ±2.83 | 4.04 |
| 10 | ±2.37 | 3.38 |
| 15 | ±1.93 | 2.76 |
| 20 | ±1.67 | 2.39 |
| 30 | ±1.37 | 1.95 |
| 50 | ±1.06 | 1.51 |
| 75 | ±0.86 | 1.24 |
| 100 | ±0.75 | 1.07 |
Your experiment protocol
Everything above, as a checklist your team can follow — run order, freeze list, and the decision rule.
3 · Ran the experiment? Upload the runs — get the verdict.
Drop your before and after Lighthouse JSON files (raw Lighthouse reports or PageSpeed Insights API exports, any number per side). The verdict comes from your runs' own measured spread — median difference, a deterministic bootstrap confidence interval, and a noise floor — per metric. Your files are parsed in memory for this session only — never written to disk, never stored, and gone when you leave the page.
Load at least one file on each side and the per-metric verdict appears here — including the consistency checks (mixed Lighthouse versions, mixed device profiles, mixed throttling) that silently invalidate most spreadsheet comparisons.
Why “just run it five times” isn't proof
Running Lighthouse five times and taking the median is excellent advice — for getting a stable single estimate of where a page sits today. It is not the same as having enough statistical power to prove a small before/after change is real. Those are different jobs, and the second one is governed by the spread of the runs:
n ≥ 2 · (z₁₋α/₂ + z_power)² · σ² ⁄ δ² (independent, equal n)
at α = 0.05 and 80% power: n ≈ 15.7 · σ² ⁄ δ²
The dependence on σ²/δ² is what bites. Halving the change you want to detect
quadruples the runs you need. On a moderately noisy page — σ ≈ 2.7 points, the middle
of the published range the presets above are derived from — proving a 3-point change needs about
13 runs per condition; a 2-point change needs 29. Median-of-five on that page can only resolve
changes of roughly five points or more — and on the noisiest sites in that study (σ ≈ 4.3) it
takes 33 runs to prove 3 points. Fine for a big regression, useless for the single-digit gains
most optimisation work produces.
What the pilot measures — and what it can't. The “measure it for me” pilot runs your page through PageSpeed Insights repeatedly and takes the per-metric standard deviation. That σ includes Google's runner mix (we report the CPU-benchmark spread when it's material), so it's the honest σ for PSI-based before/afters; a pinned local or CI setup is usually tighter, so measure there if that's where your experiment will run. A small pilot's σ̂ is itself uncertain — we show its 90% band and the conservative run count from the upper bound rather than hiding it. And note the pilot yields the run-level σ: a paired design needs the SD of paired differences from your own harness. Lab CLS is often deterministic; when the pilot measures σ = 0, any before/after CLS difference is directly interpretable and there is no sample-size question to answer.
How the upload verdict decides. The before/after comparison never trusts a single number: it takes the median of each side, builds a deterministic bootstrap confidence interval on the median difference from your runs' own spread, and sets a noise floor from the within-arm variation. A change is called real only when the interval excludes zero and the change clears the floor — and an apparent regression larger than the page's own baseline is treated as instability, not as a confident result. It also checks what spreadsheets silently miss: mixed Lighthouse versions, mixed device profiles, mixed throttling, and multiple URLs in one comparison.
The full method, the secondary evidence, and the registered protocol behind this calculator are in the study “How many Lighthouse runs prove a change?”
Free 5-day email course
Read your PageSpeed report like an auditor
The statistics behind this calculator are day 5 of the course — the other four days teach the rest of the method, every number from our published worked example:
- Day 1The 81% problem — a real report, 21 flags, and why only four deserved effort.
- Day 2The four verdicts — Real, Wrong, Trivial, Locked — and the test each flag must pass.
- Day 3Falsifying a flag — three places the disproof hides, starting with the tool's own numbers.
- Day 4Hearing the silence — the five-minute field-vs-lab drill that outranks every flag.
- Day 5Predict, prove, confirm — this calculator's math and the 28-day rule that make a fix provable.
The course lives in email only — it isn't published anywhere on the site.