One sentence, then every pull request.

Two things you already have: a URL that is reachable, and a sentence describing something that should work. Staging, a deploy preview, a localhost through a tunnel — any of them.

export ANTHROPIC_API_KEY=sk-ant-…

npx smolanalytics test \
  --url https://yourapp.com \
  --test "a visitor can sign up with an email and land on the dashboard"

The agent opens a real browser, reads the page the way a screen reader does — the actual buttons, fields and their state — decides what to click, and checks what it sees against your sentence. It never guesses at pixel coordinates, so it does not click the wrong thing and then blame the wrong feature. Add --headed to watch it happen.

The key is yours and stays yours: the agent runs on your machine, on your own Anthropic key, and replaying a recording needs no key at all. The first run also needs Playwright and Chromium (~50MB), which go in ~/.cache/smolanalytics, once. In a terminal it stops and tells you that before fetching anything — re-run with --yes to let it — and in CI, where there is nobody to answer, it installs. Nothing is written to your project either way.

Say what you expect to see. "checkout works" cannot fail usefully. Name the page, the control and the evidence, the way a careful person would check it by hand.

npx smolanalytics suggest --url https://your-staging-url.com

A real browser walks your app — same-origin pages only, reading, never clicking or submitting — and writes the flows worth testing into tests/*.md, in the format --suite already runs. At most six proposals by default (--max <n>; a small app honestly yields fewer), into tests/ or wherever --out points, and a file that already exists is never overwritten. Delete the ones you disagree with.

It only proposes what it actually saw. Every proposal has to quote text that appears on a page the crawl read; one that cannot is dropped, out loud, with the reason. A model asked “what should an app like this test?” answers from every app it has ever read about — password resets, wishlists, coupon codes — and one such file is worse than an empty folder, because it fails forever against a feature you never built. It reads the running app, not your codebase: the sentences are still yours to keep or throw away.

A suite is a directory of markdown files. One heading is one test; the prose under the heading is the whole test. That is the entire format, and it is deliberately readable by someone who does not write code.

# Checkout

## A shopper can add an item to the cart

From the storefront, open the first product, add it to the cart, and check that
the cart shows one line for that product at the price the product page listed.

## An expired card is refused with a message that says so

At checkout, pay with card number 4000 0000 0000 0069, any future-looking name,
CVC 123, and check that the page stays on checkout and says the card is expired
— not a generic "something went wrong".

## An empty cart says it is empty

Open the cart with nothing in it and check it says the cart is empty and offers
a way back to the storefront, rather than showing a total of 0 with a checkout
button.

npx smolanalytics test --suite tests/ --url https://staging.example.com

Optional frontmatter at the top of a file (title, criticality) is read and kept out of the sentence. A file that opens with one # title and lists ## flows under it is understood as a title plus its tests, not one enormous test.

Two things worth knowing before you commit the folder. The heading is the test's identity and the recording's filename, so renaming a heading throws its recording away and the agent runs that test fresh. And never put a real password or card number in there — the file is committed. Point the tests at a seeded account on staging and your provider's test card.

One workflow file, and the verdict arrives on the pull request that changed the app without anyone remembering to run anything. Copy this to .github/workflows/e2e.yml, add ANTHROPIC_API_KEY to the repository's Actions secrets (Settings → Secrets and variables → Actions), and keep one of the three "the preview URL" steps.

That is the whole install. No GitHub App across your repositories, no agent pushing a Dockerfile into your code, no environment built before you are allowed to see anything. It runs on your own Actions runner, against a URL your host already builds for the pull request, and comments with the GITHUB_TOKEN Actions hands every job for free.

# .github/workflows/e2e.yml
name: e2e

on:
  pull_request:

# THESE TWO, AND NOTHING ELSE.
#   contents: read        actions/checkout reads tests/ out of this repository.
#   pull-requests: write  the verdict is one comment, edited in place on every push. Without it
#                         everything runs and only the comment fails, with a 403.
# Naming them here also takes everything else away: a compromised step in this workflow cannot
# push code, cut a release, or reach another repository.
permissions:
  contents: read
  pull-requests: write

# One run per pull request. Push twice in a minute and two runs drive two browsers against the
# same preview and race to edit the same comment.
concurrency:
  group: e2e-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  e2e:
    runs-on: ubuntu-latest
    # A hung browser otherwise runs to the six-hour default and bills six hours of somebody's
    # minutes for a test that was never going to finish.
    timeout-minutes: 30

    # WHO DOES NOT GET SECRETS, and would therefore run every test against an empty key and then
    # 403 on the comment: a fork's pull request, and dependabot. Skipping says that plainly.
    if: >-
      github.event.pull_request.head.repo.full_name == github.repository
      && github.actor != 'dependabot[bot]'

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22

      # THE PREVIEW URL — ALL THREE OF THESE ARE OFF, AND FOR MOST REPOSITORIES THAT IS CORRECT.
      # With no step called preview, the URL below resolves to empty, --url "" reads as no --url
      # at all, and the runner finds the deployment itself through the deployments API. Uncomment
      # one only if that does not work for you.

      # (a) VERCEL builds the preview itself; this waits for it and reads its URL. Netlify and
      #     Render have equivalent actions. A third-party action inside a job holding
      #     pull-requests: write is worth a moment's thought, which is the other reason this is
      #     not the default.
      # - name: the preview URL
      #   id: preview
      #   uses: patrickedqvist/wait-for-vercel-preview@v1
      #   with:
      #     token: ${{ secrets.GITHUB_TOKEN }}
      #     max_timeout: 600

      # (b) A URL YOU ALREADY HAVE — staging, or a preview at a predictable address.
      # - name: the preview URL
      #   id: preview
      #   run: echo "url=https://staging.example.com" >> "$GITHUB_OUTPUT"

      # (c) BUILD AND SERVE IT HERE, for a static site with no preview host. Wait for the server:
      #     a test against a port nothing is listening on yet reports a broken app when the app
      #     is fine.
      # - name: the preview URL
      #   id: preview
      #   run: |
      #     npm ci
      #     npm run build
      #     npx --yes serve --listen 4173 dist &
      #     npx --yes wait-on --timeout 60000 http://localhost:4173
      #     echo "url=http://localhost:4173" >> "$GITHUB_OUTPUT"

      # THE RECORDINGS. These two steps are the economics of the whole tool. A CI runner starts
      # empty every time, so without this cache every test on every pull request is a fresh agent
      # run: slower, and paid for per run, forever. The key is per-commit so each run saves a
      # fresh entry; restore-keys takes the newest earlier one.
      - name: restore the recordings
        uses: actions/cache/restore@v4
        with:
          path: .smolanalytics/recordings
          key: smolanalytics-recordings-${{ github.sha }}
          restore-keys: |
            smolanalytics-recordings-

      # Chromium is downloaded each run and deliberately not cached: it is hundreds of megabytes
      # against a repository's 10 GB allowance, so a handful of runs would evict the recordings —
      # the part worth keeping and the part that cannot be re-downloaded.
      - name: e2e
        # WEEK ONE: REPORT, DO NOT BLOCK. A new tool that puts a red X on somebody's pull request
        # before it has earned any trust gets uninstalled instead of read. The comment is still
        # posted and the step is still marked failed in the log. Delete this line once the suite
        # has been right often enough that a failure should stop a merge.
        continue-on-error: true
        env:
          # Referenced as "$URL" below rather than pasted into the shell command, because a value
          # that came from another action must never be interpolated into a script.
          URL: ${{ steps.preview.outputs.url }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          # Provided automatically by Actions. This is the whole of the GitHub access this needs.
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: npx smolanalytics@latest test --suite tests/ --url "$URL" --comment

      # SAVED AS ITS OWN STEP, AND if: always(). The all-in-one actions/cache saves only on
      # success — invisible until the day you delete the continue-on-error above, and from that
      # day a run with one failing test throws away every recording the agent repaired in it.
      # The failing run is the run that re-recorded the most.
      - name: save the recordings
        if: always()
        uses: actions/cache/save@v4
        with:
          path: .smolanalytics/recordings
          key: smolanalytics-recordings-${{ github.sha }}

The comment is one comment, edited in place: a headline count, a table of every test with how it ran (agent, or replayed with no model calls) and how long it took, and the failure text in full underneath, because the person reading it has not been watching the browser. Twenty pushes do not mean twenty comments. When a test failed, the comment also names the changed files most likely responsible and groups the failures that share a cause — see when a test fails.

On a big folder, --since <ref> runs only the tests this change could have broken and says in the headline what was skipped. Anything it cannot rule out still runs, and a skipped test is never counted as a pass.

Exit codes, if you gate on this later: 0 every test passed, 1 a test failed — the app did not do what the sentence describes, 2 this runner could not finish (no browser, no key, no network, or a recording stopped fitting and there was no key to re-check it with). Never your application.

Set two more environment variables and every verdict also lands in your project, under "Did anything break?" on the project page: SMOLANALYTICS_PROJECT (the project id) and SMOLANALYTICS_WRITE_KEY. The write key is public and can only add, never read, which is why the runner can carry it in CI. If that post fails the verdict above it still stands: telemetry never changes an exit code.

Five verdicts, and they are never blurred into each other. Blur them and a copy change pages somebody at 2am, or an outage on our side reads to you as your checkout being broken.

A run that passes is recorded, and the recording replays with no model calls at all. Measured against this site: the first run took 8.0s with the agent, the second 1.4s with no model. Those are one flow on one machine rather than a promise about yours, but the shape holds — the agent only re-engages when the recording stops fitting the app.

# first run: the agent, and it writes the recording
npx smolanalytics test --url https://yourapp.com \
  --test "the pricing page shows a monthly price" --plan pricing.json

# every run after: the recording, no model, no key needed
npx smolanalytics test --url https://yourapp.com \
  --test "the pricing page shows a monthly price" --plan pricing.json

For a suite, recordings live in .smolanalytics/recordings (change it with --plans <dir>), one JSON file per test, named from the file it lives in and its heading. In CI they must be cached or every pull request pays the agent for every test, forever — which is what the two cache steps in the workflow above are for.

A failed verdict on a pull request comes with the changed files most likely responsible. No model call: it is the intersection of what the failing run observed — the controls it clicked, the paths it visited, the proof text that vanished — and what the pull request changed, read from git's own diff against the base branch.

src/Checkout.tsx — this PR removed the string 'Proceed to checkout' this test clicks

Two rules shape it. No suspicion without named evidence: every line says which string or path connects the file to the test, and a file it cannot connect is never mentioned. Zero matches means nothing at all: “fourteen files changed, one of them probably did it” is the whole diff ranked as vaguely suspicious, so when no changed file touches anything the test used, no suspect lines appear. A blame hint never changes a verdict or an exit code, and if there is no git, no base ref or the repository is too big to diff, it degrades silently to no suspects.

Twelve red tests are usually one thing. Failures are grouped by what they share: the same suspect file, each blamed independently for its own reason; the same control in both recordings (a real click on the same accessible name, not anybody's description of the failure); or, one level weaker, the same path. Never by the failure prose — two failures that both say “the page showed an error” are not evidence of a shared cause, and grouping on it is how a real second bug gets filed inside a group somebody dismissed as all one thing.

Why it was flaky. A flaky verdict carries both runs — every step, every target, every step's duration — compared step for step, with no model call. When the two differ in a way that names a race in your app, the reason says so with the evidence; when they cannot be told apart it says exactly that, because a wrong diagnosis sends somebody hunting a race that does not exist, or tells them to ignore one that does.

The bug is already written, so it can be filed. A failure here is a sentence naming the page, the control, what was expected and what the page did instead, plus the suspect and its evidence. Set the variables for your tracker and every failed test becomes one issue:

# Linear
SMOLANALYTICS_LINEAR_API_KEY=lin_api_…
SMOLANALYTICS_LINEAR_TEAM_ID=…

# Jira
SMOLANALYTICS_JIRA_URL=https://yourcompany.atlassian.net
SMOLANALYTICS_JIRA_EMAIL=you@yourcompany.com
SMOLANALYTICS_JIRA_API_TOKEN=…
SMOLANALYTICS_JIRA_PROJECT=ENG

One issue per test, however many pushes it fails on: each carries a fingerprint of the test it came from, and an existing open issue with that fingerprint is updated rather than duplicated. Only failed is filed. stale is our recording aging, flaky is a ticket everybody closes as cannot-reproduce, and errored is our runner — filing any of them against your product is a lie about whose fault it is.

Telling somebody who is not looking at the pull request. Set SMOLANALYTICS_SLACK_WEBHOOK and the verdict lands in Slack (SMOLANALYTICS_WEBHOOK posts the same as JSON anywhere else). What is sent is the ship verdict, not a count — “Do not ship this” and the causes, rather than “3 failed” — and by default it speaks only when something failed or went unverified, because a channel that says “all good” on every push is a channel that gets muted. The URL is a bearer credential and is read from the environment only, never a flag. Like the tracker and the comment, a delivery that fails can never reach the exit code.

npx smolanalytics guard          # this repo; or: npx smolanalytics guard ./api --json

An environment variable the code reads with no fallback and names in no .env.example, README, Dockerfile, compose file or workflow is a deploy that starts and then fails on the line that reads it. This reads the repository it is standing in and lists those: no key, no account, no network, no config.

A read with a fallback is not a requirement. process.env.PORT || 3000 cannot break a deploy and is never reported as if it could, because flagging optional configuration is how a check becomes noise and then gets deleted. --json is for a CI step that wants to act on it.

Most tests worth writing are behind a login, so the sign-in is recorded and reused exactly the way everything else here is.

export SMOLANALYTICS_LOGIN_EMAIL=qa@yourcompany.com
export SMOLANALYTICS_LOGIN_PASSWORD=...

npx smolanalytics test --suite tests/ --url "$URL" \
  --login "sign in as {{email}} with {{password}}"

The agent signs in once for the whole suite, and every test after that starts already signed in — measured at fifty tests, one sign-in. If the session expires part way through, it signs in again, once, and carries on.

The password is read from the environment and never written down. The recording stores {{password}}, not your password, and resolves it at the moment of the keystroke — so a recording you commit, or a CI cache somebody restores, carries the shape of your login and nothing else. The saved session lands in .smolanalytics/auth/, which is given a .gitignore of its own the first time it is written, because that file holds a live session cookie.

If a sign-in does not work, that is reported as errored — our side — and never as a failed test. A wrong password says nothing about whether your product works, and a red X on a working login is worse than no test at all.

Already generate a Playwright storage state? --auth-file path/to/state.json uses it and skips all of this.

A test’s proof is text on the page. That means a build whose CSS failed to load, that rendered blank, or that is showing a crash overlay with the text still in the DOM would otherwise pass — a green tick on a page nobody could use. So a passing run also checks that the page actually rendered:

The steps all worked and the page still says what it should — but a
full-viewport error surface is covering the page: <div#o> opens with
"Unhandled Runtime Error"

It fires only on catastrophe: a blank viewport, stylesheets that failed to load, a framework error surface. A canvas game with no text, an image-only gallery, a dark theme, a cookie banner covering the whole viewport and an app that paints half a second late are all left alone — a check that cried wolf would cost more trust than the bug it caught. --no-render-check turns it off.

Controls inside same-origin iframes and open shadow roots are read and can be acted on, and a step says which document it happened in, so a click inside an embedded checkout is not reported as a click on your page. A closed shadow root genuinely cannot be read from outside, and the run says so rather than reporting the page as empty — the failure worth avoiding is an agent confidently calling a working embedded checkout broken.

For flows that create data, write {{email}}, {{password}} or {{runid}} in the sentence. Each run gets its own identity on an address that cannot receive mail, prefixed smoltest so one LIKE 'smoltest%' finds everything a test ever made. Add --teardown <url> and that identity is POSTed to an endpoint of yours after every run — including the runs that failed — so your app can delete what the test created.

The same walk through your product knows which user actions exist. That is what makes the second half possible, and it is the half nobody else has: we write and maintain your analytics tracking inside the SDK you already use — Google Analytics, PostHog, Plausible, Mixpanel, Amplitude or Segment. We do not replace your analytics. We keep its instrumentation correct.

Start with the free one. It needs no account, no key, and makes no network call, so your code never leaves your machine:

npx smolanalytics audit

It reads the repo you are standing in and names the user actions nothing is measuring, with the file and the line — payments first, then signups, logins, invites. Then your coding agent writes the calls over MCP (propose_instrumentation hands back the exact call for each site, in whichever SDK the repo already runs), and verify_instrumentation answers FIRING, WIRED or MISSING per event rather than making you wait a day to find out.

When an event in your tracking plan stops arriving but the traffic behind it doesn't, smolanalytics finds the commit that deleted the track() call and opens a pull request putting it back. The line comes back verbatim out of your own git history, in your own SDK, and it is off by default twice over: a deployment-wide switch, and a per-project setting that starts off.

The gate for CI, once a plan exists:

npx smolanalytics plan check --project <name> --window 24

It fails the step when an event your tracking plan declares has stopped firing, so a refactor that deleted a track() call is caught on the pull request rather than a week later in a report. --window <hours> counts only recent events; --key and --url stand in for the environment variables.

Already running PostHog? Connect it from your project's setup page: paste a read-scoped key, we prove it against a real query before storing it, and the instance reads it as bounded aggregate queries on a schedule. Writing a tracking call needs only your repo, so we write for all six SDKs above; telling you a call stopped firing needs the numbers, and today we can read PostHog or our own ingest. The rest of the connections are on the integrations page, and the whole instrumentation loop has a page of its own.

The same runner is an MCP server over stdio, so the agent you already write code with can run the suite and read every verdict without leaving the editor:

{ "command": "npx", "args": ["smolanalytics", "mcp"] }

Three tools: run_tests runs the suite and hands back every verdict, can_i_ship is the verdict plus what was not checked, and list_tests is what the folder already covers. It is local only — it never comments on a pull request, never publishes a share link, and never records a run to a project — and it needs the same ANTHROPIC_API_KEY the terminal does.

The third of those is the hosted half, and it is one more command. It writes the entry for your organization's MCP endpoint (smolanalytics.com/api/mcp, the API token from Settings) into every assistant it finds installed, or just one:

npx smolanalytics connect                 # cursor, claude-code, vscode, windsurf, claude-desktop, cline
npx smolanalytics connect cursor --key <org token>

One token operates every project: the agent provisions a new one with create_project and reaches any of them by passing project="<name>", with the read key kept server-side. What it does there is the next section. Every editor's exact config is on the MCP page.