Flaky test
what is flaky test?
A flaky test is one that passes and fails on the same code, with nothing changed between the two runs. It is not a bug report and not a pass: it is a measurement you cannot trust. Flakiness is the leading cause of teams muting an end-to-end suite, because a red build that nobody believes is worse than no build at all.
Four causes cover most of it. Timing, where the test checks for something the page has not finished drawing. Shared state, where two tests running in parallel use the same account or the same row. External dependencies, where a third-party script or a payment sandbox answers slowly. And selector fragility, where the test is bound to a class name or a DOM position that changes for reasons unrelated to behaviour.
The expensive part is not the failed run, it is what a team does next. A suite that cries wolf gets rerun until it is green, then gets marked as allowed to fail, then gets muted. At that point the tests still exist, still cost money to run, and catch nothing, which is worse than having none because it looks like coverage on a dashboard.
The fix that works is not more retries. It is removing the reason a run can differ: wait for the state rather than the clock, give each test its own data, and stop binding tests to selectors that describe how a page is built rather than what it does.
in smolanalytics
Flaky is one of five verdicts and it is never folded into a pass or a failure. A test that fails and then passes on a retry within the same run is reported as flaky, named as such in the pull request comment, and counted separately. The reason for the separate word is that a flaky result is a fact about the test, and a failure is a claim about your product, and blurring the two is exactly how a suite gets muted.
How it works shows where this fits in the loop, and the docs have the exact behaviour.