glossary

The words this category uses, defined plainly.

Each definition is written to be true for the whole category, not for one product. Where smolanalytics handles something in a particular way, that is a separate paragraph at the end of the page, labelled.

Flaky test: A flaky test is one that passes and fails on the same code, with nothing changed between the two runs. It is not a bug report and not a pass: it is a measurement you cannot trust. Flakiness is the leading cause of teams muting an end-to-end suite, because a red build that nobody believes is worse than no build at all.
Stale recording: A stale recording is a recorded test run that no longer fits the application: a control it clicked cannot be found, or a step no longer applies. It is not a failure. A rename and a regression look identical to a replay, so reporting stale as a bug would mean claiming a product is broken every time somebody renames a button.
Tested pull request: A tested pull request is one pull request on which at least one test ran, counted once however many times the suite ran on it. It is a billing unit rather than a technical one, and it exists so that re-running a suite after a fix costs nothing extra, which the common alternatives, credits and runner minutes, all quietly penalise.
Deterministic replay: Deterministic replay means re-running a test from a recorded sequence of steps rather than by reasoning about the page again. Because no model call is made, the run is fast, cheap and identical every time. The trade-off is that a recording is a snapshot: when the page moves, the replay stops fitting and something has to decide what happens next.
Accessibility tree testing: The accessibility tree is the structured representation a browser exposes to screen readers: the role, name, value and state of every element on a page. An agent that drives a browser through this tree picks a named control rather than a screen coordinate, which is why it does not click the wrong thing when a layout shifts or a page renders at a different size.
Self-healing tests: A self-healing test repairs itself when the interface changes, rather than failing because a selector moved. Implementations differ enormously: some retry with alternative selectors, some ask a model to re-find the element, and some regenerate the test file and open a pull request. The question worth asking of any of them is what happens when the healing is wrong.
End-to-end test: An end-to-end test exercises a complete user journey through the real system: a browser, the application, the database and whatever third parties are involved. It answers whether a person can actually do the thing, which no unit or integration test can, and it is the most expensive kind of test to keep alive because everything it touches can change underneath it.
Agentic testing: Agentic testing means an autonomous agent decides how to exercise an application, rather than replaying steps a person wrote down. You describe the outcome; the agent reads the page, chooses actions and judges the result. It removes selector maintenance and adds two new questions: what a run costs, and whether the agent's judgement of success can be trusted.
Vibe testing: Vibe testing is the practice of verifying software that was generated rather than written line by line, by checking that user journeys still work instead of reviewing the code that changed. It is the counterpart to vibe coding: when changes arrive faster than anyone reads them, behaviour becomes the only thing left worth checking.
Preview environment testing: Preview environment testing means running tests against a deployment built from a single pull request, rather than against a shared staging site. It tests exactly the change under review with nobody else's work mixed in. The cost is that somebody has to build and seed that environment, which is real infrastructure work with real recurring expense.
Test suite maintenance: Test suite maintenance is the continuing work of keeping tests accurate as the product changes: updating selectors, re-recording flows, fixing data assumptions and investigating failures that turn out to be about the test rather than the product. It is the dominant lifetime cost of end-to-end testing, and it is almost never budgeted, which is why suites are abandoned rather than deleted.
CI test gate: A CI test gate is a check that blocks a pull request from merging when tests fail. Its usefulness depends entirely on precision: a gate that also blocks on infrastructure problems and unreliable tests gets bypassed within weeks, and a bypassed gate is indistinguishable from no gate.