how to analyse AI agent conversations without paying per message

run the analysis on the model you already pay for. connect smolanalytics over MCP, have your model call sample_conversations to read a sample of real conversations, infer labels like intent, sentiment and frustration, then write each one back with label_conversation. the counts per label value are computed from your event log. nobody runs an LLM over every message on your behalf, so there is nothing to meter, and it is free and unmetered on every plan.

why per-message analysis gets expensive

if a vendor promises to tell you what people are asking your agent about, something has to read the conversations. in the usual product that something is an LLM the vendor runs on their side, over every message that arrives. that is a real cost to them, so it becomes a line on your bill, priced per message or per session or as a separate "AI" meter.

the shape of that bill is the problem, not the price. it scales with usage, which is exactly the thing you want to grow. every extra conversation your agent handles makes the analysis more expensive, and the marginal value of the ten-thousandth labeled conversation is close to zero, because you already knew the split by the two-hundredth. you end up paying linearly for information that stopped arriving a long time ago.

there is a second cost that never shows up as a number. to run that analysis, the vendor has to receive and process the full text of your users' conversations. for a lot of apps that is the whole reason conversation analytics never got turned on.

the arithmetic

take a small but real agent: 50,000 conversations a month, 8 turns each, roughly 120 tokens a turn. that is about 1,000 tokens of conversation text per conversation, 50 million tokens a month in total.

now compare reading all of it to reading a sample of 200 conversations a week, which is 800 a month.

the cost column uses a mid-tier model rate of about $3 per million input tokens and $15 per million output tokens, which is roughly where a Sonnet-class model sits. that is the raw token cost of the work, before anyone adds margin, infrastructure, or a per-message price. the ratio is what matters: 62x fewer tokens for the same answer, and the sampled column is spend on your own model account that you already have open.

then double your traffic. the top row doubles. the bottom row does not move, because a sample of 200 conversations a week is still 200 conversations a week. that is the actual difference between the two models, and it is why this is free and unmetered on every smolanalytics plan: it costs us nothing, because we never run it.

is a sample actually enough?

this is the fair objection, and it is answerable with arithmetic rather than opinion.

you are estimating a proportion: what share of conversations are about billing, what share look frustrated. the uncertainty on a proportion depends on how many things you looked at, not on how many things exist.

label 200 conversations and a category that comes back at 30% is somewhere around 23% to 37%. that is completely sufficient for "billing questions are the biggest bucket and support is second", which is the decision you are actually making. it is not sufficient for "billing went from 30.4% to 31.1% this week", and nobody should be making decisions on that anyway.

labels are append-only events, so the sample compounds. label 200 a week and by the end of a quarter you have a few thousand labeled conversations sitting in the log, all still counted. you never re-pay to keep them.

the loop, exactly

four calls, all over the one MCP connection. no pipeline, no export, no second data store.

1. send turns. your app emits an agent_turn event per turn with conversation_id, role, and timing. include a text property if you want the model to be able to read what was said. if you do not send text, a turn comes back with role and timing only and has_text false, because nothing is ever invented to fill the gap.
2. sample. your model calls sample_conversations. it returns whole conversations, turns oldest first, most recent conversation first, capped (20 by default, 100 max) and deterministic: same data in, same sample out. each conversation carries a labeled flag so the model can skip ones already done.
3. infer and write back. for each conversation the model decides on labels and calls label_conversation with the conversation id, a small labels object like {"intent":"billing","sentiment":"negative","frustration":"high"}, and labeled_by set to its own model name. labeled_by is required. nothing is mutated; one new agent_label event is appended.
4. count. ask agent_labels (or GET /v1/agent/labels?label=intent) for conversations per value of a label. the counts are computed from the event log the same way every other number here is, and the result always carries which models wrote the labels plus how many conversations in the window are labeled and unlabeled.

the whole thing is one prompt in your editor: "sample 50 conversations from last week, label them by intent and sentiment, then show me the intent breakdown". your model does steps 2 and 3 in a loop and reads step 4 back to you. more on how the connection itself works on the MCP page, and there is a plain definition of the pattern in the analytics MCP server glossary entry.

what you get back, and what it honestly is

the counts are computed. the labels are that model's inferences. those are different kinds of thing and the reports never blur them: every label breakdown names the model that wrote the labels, and shows labeled and unlabeled counts beside the split, so a guess is never presented as a measurement. if nothing has been labeled yet you get an honest empty result telling you to sample first, not a fabricated split.

so when you read "38% billing, 22% onboarding", the correct reading is: of the 200 conversations claude-sonnet-4-5 looked at, it judged 76 to be about billing. the 76 is a fact. the judgement that those 76 were about billing is a model's opinion, a good one, and still an opinion. use it to rank and to decide where to look, not as evidence in an argument. i wrote more about where that line sits in computed vs inferred.

three more limits worth stating plainly. a sample is not a census, so rare categories under a few percent will not show up reliably. label quality is your model's quality, and inconsistent label names fragment the breakdown, so keep the vocabulary small and reuse it. and the model can only read text your app actually sent.

the rest of the picture

conversation labels sit next to the computed side of agent observability: tool-call counts, error rates, latency p50/p90/p99, client split, error taxonomy by type, and conversation health (turns, re-ask rate, abandon rate, time to first token). those are all computed from your events, with a CI test pinning /v1 and MCP and the dashboard to the same answer. what to actually measure goes through them one at a time, and the features page lists the rest of what ships in the binary.

it is the same single go binary, MIT, that does your web and product analytics too, so an agent app gets both from one install. if you are building on top of a model, the AI apps page is the shorter version of this.

cost, plainly

conversation intelligence is free and unmetered on every plan, including self-host, because it runs on your model and never on ours. what you pay for is events: Pro is $49/mo for 1M events, Scale is $149/mo for 10M, extra millions are $8 each. 14-day full trial, no card, no free cloud tier. self-hosting is free forever and always will be. full breakdown on pricing.

if you want to see how this compares to the other analytics MCP servers out there, that roundup is here. the honest summary of this particular feature: it is not magic, it is a sample plus your own model plus arithmetic you can check. that is the whole reason it can be free.