Test suite maintenance
what is test suite maintenance?
Test suite maintenance is the continuing work of keeping tests accurate as the product changes: updating selectors, re-recording flows, fixing data assumptions and investigating failures that turn out to be about the test rather than the product. It is the dominant lifetime cost of end-to-end testing, and it is almost never budgeted, which is why suites are abandoned rather than deleted.
Writing the test is the visible cost and the small one. The invisible cost arrives every week afterwards, in ten-minute increments, when somebody has to decide whether a red run means anything.
That decision is the expensive part. Each investigation costs context-switching from whoever is on call for the build, and most of them end in "the test was wrong", which teaches everyone that red means nothing.
Suites therefore rarely get deleted. They get muted, marked allowed-to-fail, or excluded from the default job, which preserves the appearance of coverage while removing all of its value. A tool that reduces authoring cost but not maintenance cost has optimised the cheap half.
in smolanalytics
There is no test code, so the maintenance that usually dominates, updating selectors after an interface change, does not exist here. What remains is deciding whether a sentence still describes what should happen, which is a product question rather than a technical one. When a recording stops fitting, it is re-derived automatically rather than left for a person.
How it works shows where this fits in the loop, and the docs have the exact behaviour.