conversation intelligence: which numbers are computed and which are a model's opinion

a computed number comes from counting events with code: same data in, same number out, every time. an inferred label is a model's judgement about what a conversation meant. counting how many conversations carry the label "frustrated" is computed. deciding which ones deserve that label is inferred. both are useful. printing them in the same font, with no marker of which is which, is how analytics tools mislead people.

the two kinds of number

almost everything in an analytics product is one of these, and most products never tell you which.

computed. 4,812 conversations last week. p90 tool latency of 1.4 seconds. a 6.2% error rate with 71% of those being timeouts. an 11% abandon rate. these come from counting and sorting events. you can rerun them and get the same answer. two different implementations of the same definition will agree. if someone challenges the number, you can walk them back to the rows.

inferred. 38% of conversations were about billing. sentiment is trending negative. users are frustrated with onboarding. these came from something reading the text and forming a view. rerun it with a different model, or the same model on a different day, and you can get a different answer. there are no rows to walk back to, only a judgement.

the trap is that inferred numbers look more valuable. "38% billing" answers a question an executive actually asked. "p90 latency 1.4s" does not. so inferred numbers get promoted to the top of dashboards and into slide decks, and by the third retelling nobody remembers a model made them up.

a working test

if you are unsure which one you are holding, ask: could two competent engineers, given the same event log and the same written definition, independently produce this number and get the same answer?

notice where the line falls in the last three rows. the count over labels is computed. the labels themselves are not. that is the whole distinction, and it is why conversation intelligence can be honest at all: you separate the judgement step from the counting step and you are explicit about which is which.

re-ask and abandon are worth a note, because they look like judgements and are not. "the conversation ended on a user turn" is a fact about turn order. calling it abandonment is an interpretation, so the metric is defined structurally, the definition is published, and you decide what it means. that is different from a model reading the ending and declaring the user gave up.

how tools end up misleading people

nobody sets out to lie. it happens in four ordinary steps.

1. a model reads the conversations and produces topic and sentiment labels. reasonable so far.
2. the labels get aggregated into percentages. the percentage is real arithmetic over unreal inputs, and percentages carry an air of precision that judgements do not.
3. the provenance drops off. the dashboard says "38% billing". it does not say which model decided, when, on how many conversations out of how many, or how many were never looked at.
4. someone acts on it. a roadmap gets rewritten around a number that was one model's read of 200 conversations, presented with the same confidence as the event count next to it.

step 3 is where the damage is done. steps 1 and 2 are fine if step 3 does not happen. the fix is not to stop inferring, it is to never let the inference travel without its provenance attached.

how to keep them apart

these are the rules the label reports here follow, and they are the rules i would apply to any tool.

1. name the model on every inferred number. labeled_by is a required field when a label is written, and it must be the model's own name. it shows up on every breakdown. an unattributed inference is not a number, it is a rumour.
2. always show the denominator, including the part you did not look at. every label breakdown carries how many conversations in the window are labeled and how many are unlabeled. "38% of 200 labeled, out of 4,812 total" is a fact. "38%" on its own invites the reader to assume it means all of them.
3. never fabricate an empty result. if nothing has been labeled yet, the report says so and tells you to sample and label first. it does not invent a plausible split. same reason resolution_rate stays null until your app sends a resolved bool: a missing number is honest, an estimated one that looks measured is not.
4. keep the inference append-only. a label is a new agent_label event, never a mutation of the underlying conversation. the raw record stays exactly as it was, so a bad labelling run is recoverable and re-labelling is just more events.
5. do not let a model write the aggregate. the model's job ends at "here is my label for this conversation". the counting is done by the same deterministic code that computes everything else. a model never gets to compute a percentage, because that is the step where a wrong number becomes an authoritative one.
6. put them in different columns, visually. if the trend chart and the model's opinion sit side by side with identical styling, the reader will treat them identically. that is a design decision, not an accident.
7. use inferred labels to decide where to look, not to prove a point. "billing is the biggest bucket, go read those conversations" is exactly what a label is for. "billing grew 0.7 points" is not something a sample of judgements can support.

why we do not run the model for you

there is a practical reason and a trust reason, and they point the same way.

the practical one is money: analysing every message server-side is a cost that scales with your traffic and lands on your bill. running it on a sample with the model you already have is roughly two orders of magnitude fewer tokens for the same answer, which is worked through with real arithmetic in analysing conversations without paying per message. it is free and unmetered on every plan because it costs us nothing.

the trust one is this: if we ran an LLM over your conversations, we would be the ones producing the judgement and asking you to accept it. instead your own model, connected over MCP, reads a sample with sample_conversations, writes its inferences back with label_conversation, and we do arithmetic over what it wrote. we never see a judgement we made. the text never has to leave your model. and you can point a different model at the same log and compare, which is not a thing you can do with a black box.

the same principle, everywhere else

this is not a policy invented for conversation labels. it is the thing the whole product is built on. the computed reports are pinned so that /v1, the MCP tools, and the dashboard return the same answer, and a CI test fails the build if they diverge. a model connected over MCP reads those reports, it does not write queries against your data, so it can hand you the real figure or nothing, never one it composed. that argument is spelled out against the other analytics MCP servers in this roundup, and the mechanics of the connection are on the MCP page.

the same line runs through the computed side of agent observability, covered in what to actually measure, and through the rest of what ships in the binary on the features page.

open source, MIT, one go binary. self-host free forever, or hosted at $49/mo for 1M events with a 14-day full trial and no card, priced here.

the short version: count what you can count, be explicit about what you guessed, and never let the second one wear the clothes of the first.