End-to-end tests for every preview on Vercel.
The hard part of end-to-end testing in CI is usually getting a running copy of the app. On Vercel you already have one per pull request. The whole setup is passing that URL to a command.
- name: end-to-end tests
run: npx smolanalytics test --suite tests/ --url "$URL" --comment
env:
URL: ${{ needs.deploy.outputs.preview-url }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}It runs on your own runner and comments with the token Actions already gives every workflow. There is no GitHub App to install.
how do I write end-to-end tests for a Vercel app?
Vercel builds a preview deployment for every pull request, which means the expensive part of end-to-end testing in CI is already done: you have a running copy of the app at a URL. The setup is passing that URL to a command — npx smolanalytics test --suite tests/ --url "$URL" --comment — in a workflow step, with the suite being a folder of markdown files containing one sentence each. An agent drives a real browser, decides what to click by reading the accessibility tree, and posts one comment on the pull request saying what broke, edited in place rather than added to. It runs on your own GitHub Actions runner and comments with the GITHUB_TOKEN Actions provides free, so there is no GitHub App to install and nothing is written to your repository. This catches the class of bug a preview exists for and rarely surfaces: an environment variable set in Production and not Preview, a redirect that only fires on the production domain, an API route that worked locally off an uncommitted .env. If deployment protection is enabled, supply a bypass token, or the run reports errored — our runner could not reach the app — rather than claiming your app failed. Passing runs are recorded and replay with no model call at all. 14-day trial at Pro limits, no card, then $19/month.
This is the shortest setup of any stack, because the expensive half already exists. Vercel builds a deployment for every push and gives you its URL; the runner takes a URL and a folder of sentences. Nothing needs provisioning, no environment has to be spun up and torn down, and there is no preview environment for us to build — which is worth saying plainly, because that is the part other tools ask you to hand over. The comment goes on the pull request from your own runner, using the GITHUB_TOKEN that GitHub Actions provides to every workflow for free, and it is one comment edited in place rather than a new one on every push.
What a preview URL is uniquely good at catching is the environment-shaped bug: the variable set in Production and not in Preview, the redirect that only fires behind the production domain, the API route that works locally because it reads from a .env file nobody committed. Those never show up in local testing by construction, and they are exactly what the preview exists to surface — if anything actually uses it, which is the part that normally never happens.
One note on protection: if deployment protection is on, the preview needs a bypass token in the environment, or the runner gets the login wall and reports errored rather than pretending your app is broken. That distinction is deliberate — errored means our side could not run, never that your app failed.
The instrumentation half rides along on the same walk. The agent has just used the flows on the preview, so it knows which user actions exist, and it writes and maintains the tracking calls in the SDK you already run — PostHog, Mixpanel, Amplitude, Google Analytics, Plausible or Segment.
How it works walks the whole loop from the sentence to the comment on the pull request, instrumentation covers the second half — the tracking calls the same walk writes and maintains — and the free tools run without an account. The docs have the flags, the suite format and the CI step.
Pro $19/mo, 100 tested pull requests included, unlimited projects, after a 14-day trial at Pro limits with no card. full pricing, including overage →
One sentence, on your Vercel app, tonight.
Point it at a URL that is already running and describe what should work. The first run uses the agent; once it passes it is recorded, so every run after that replays with no model at all — measured on our own site, 8.0s the first time and 1.4s the second.