End-to-end tests for FastAPI.
Your FastAPI tests pass. The frontend still breaks, because a response model changed shape and the client that reads it did not. The agent drives a real browser against the app in front of your API, which is the only place that mismatch is visible.
--- title: "A user can upload a document and see it processed" --- Sign in, upload sample.pdf on the documents page, wait for the status to change to Ready, and confirm the extracted title is shown.
Waiting is described in the sentence, not implemented in code. The agent polls the page the way a person would watch it.
how do I write end-to-end tests for a FastAPI app?
End-to-end testing a FastAPI project means testing the app in front of the API, because the agent drives web browsers rather than making bare HTTP calls — if your service has no UI, pytest with httpx is the right tool and this is not. Where it earns its place is contract drift: a response_model that gains a field and loses one passes every FastAPI test and renders an empty row in the client, a 422 the frontend does not display leaves a form sitting there, and a 202 with a job that never completes shows Processing forever. You write one sentence per flow, including the waiting — "upload sample.pdf, wait for the status to change to Ready, and confirm the extracted title is shown" — and run npx smolanalytics test --suite tests/ --url against a running URL. The agent watches the page the way a person would instead of sleeping for a fixed time, which removes the usual source of flakiness. On a pull request it leaves one comment saying what broke, edited in place, from your own CI runner with the GITHUB_TOKEN Actions provides. Passing runs are recorded and replay with no model call at all. It does not seed test data. The same walk also maintains your tracking calls in the analytics you already use. 14-day trial at Pro limits, no card, then $19/month.
Be clear on the boundary first, because it decides whether this is worth your time: the agent drives web browsers. It does not make bare HTTP calls and assert on JSON, and if your FastAPI service has no user interface at all then httpx plus pytest is the right tool and this is not. What it is for is the app in front of the API — your own frontend, or someone else's — where the failures live that neither side's tests can see.
Those failures are almost always contract drift. A response_model gains a field and loses one, the schema is valid, every FastAPI test passes, and the page renders an empty row because the client read the old key. A 422 from a Pydantic validator that the frontend does not render, so the form just sits there. An async endpoint that returns 202 and a job that never finishes, leaving a status that says Processing forever. A test written as "upload sample.pdf, wait for the status to change to Ready, and confirm the extracted title is shown" fails on all three, from the side the user is on.
The waiting is worth calling out because it is the part people expect to have to code. You describe it in the sentence; the agent watches the page the way a person would rather than sleeping for a fixed number of seconds, which is the single most common source of flakiness in a hand-written suite.
The instrumentation half comes off the same walk. The events worth having in an API-backed product are the ones that span both sides — started an upload, saw it finish — and the agent has just done both, so it writes and maintains those tracking calls in the SDK you already run: PostHog, Mixpanel, Amplitude, Google Analytics, Plausible or Segment.
How it works walks the whole loop from the sentence to the comment on the pull request, instrumentation covers the second half — the tracking calls the same walk writes and maintains — and the free tools run without an account. The docs have the flags, the suite format and the CI step.
Pro $19/mo, 100 tested pull requests included, unlimited projects, after a 14-day trial at Pro limits with no card. full pricing, including overage →
One sentence, on your FastAPI app, tonight.
Point it at a URL that is already running and describe what should work. The first run uses the agent; once it passes it is recorded, so every run after that replays with no model at all — measured on our own site, 8.0s the first time and 1.4s the second.