What would it have told you, and on which day?

Microsoft's published measurement across thousands of controlled experiments found roughly a third of tested ideas improve their target metric; Google and Netflix have put it closer to one in ten. Pendo's telemetry across 615 subscriptions found 80% of features are rarely or never used. A replay over any real product's last quarter will find something, because the base rate guarantees it.

A dashboard says “here is your product”. A replay says “on 14 June this would have told you checkout was broken” — and you know exactly when you actually noticed. That second date is the one nobody else can produce, and it is the honest measure of what the tool is worth to you specifically.

Findings need enough data to become conclusive, so most arrive a few days after the change itself. We print that number rather than hide it, because “we'd have told you the same day” is usually false and a sceptical reader checks the least believable claim first. Four days late still beats the six weeks it usually takes someone to notice on their own.

Findings render in the order they would have arrived, and the total is stated. A regression visible on forty consecutive daily sweeps is one thing you would have been told about once, not forty — deduplicated on the metric and the change day, keeping the earliest sweep that saw it.

Each sweep receives only events strictly before its own date; deploys and experiments are filtered the same way. Testing only that a future ship is never named would pass just as well if attribution were broken outright — so the control case is tested too: a ship that landed on the change day must be named. Writing that control is how we found that ship attribution had never fired for anyone, because a threshold measured in fractions had been passed a whole number.