agent observability: what to actually measure

six numbers cover most of it. per tool: call count, error rate, and latency p50/p90/p99. across tools: an error taxonomy by error type. per conversation: re-ask rate, abandon rate, and time to first token. plus resolution rate, which only exists if your app says whether the thing got resolved. all of those are computed from events. anything about intent or sentiment is a model's inference, and belongs in a separate column.

the thing most agent dashboards get wrong

most agent observability starts as tracing, which makes sense, because when an agent misbehaves you want to see the exact run. traces are excellent for debugging one thing and bad at answering "is this getting worse". then token counts become the headline, because tokens are easy to count and map to a bill. tokens tell you what the agent cost, not whether it worked.

what you want is a small set of aggregates that move when the product breaks. below is the set i settled on, and why each one earns its place.

the six, and one that needs your help

everything above the line comes out of two event types. agent_tool_call carries the tool name, whether it errored, the error type, latency in ms, and which client made the call. agent_turn carries a conversation id, a role, and timing, including ttft_ms. that is the whole schema. the reports are agent_tools, agent_errors and agent_conversations over MCP, or GET /v1/agent/tools, /v1/agent/errors and /v1/agent/conversations over HTTP, and a CI test fails the build if the two ever disagree.

latency percentiles, not averages

average latency is the number that hides the outage. if a tool answers in 80ms for 95 of 100 calls and hangs for 30 seconds on the other 5, the average is about 1.6 seconds, which looks vaguely bad and tells you nothing. the percentiles tell you the truth: p50 80ms, p99 30 seconds. one of those numbers is the product working. the other is the reason someone is writing you an angry message.

for agents this matters more than for a web backend, because a slow tool call does not fail cleanly. it stalls mid-conversation while the user watches a cursor blink. p90 and p99 are where that lives, and they are the numbers to alert on.

the client split in the same report is underrated. a tool can be fine from one client and terrible from another, usually because of how the client batches or retries. seeing cursor, claude-code and copilot broken out separately turns "the tool is slow" into "the tool is slow from one client", a different and much smaller bug.

error taxonomy beats an error count

"7% error rate" is not actionable. 7% split into timeout 4.8%, invalid argument 1.9%, rate limited 0.3% is three separate tickets with three different owners, and you can see immediately which one to take.

this is why errors get their own report broken down by error_type rather than living as a single column. you can scope it to one tool when you already know where the trouble is. an unseen tool returns an honest empty taxonomy rather than a fake zero, which matters more than it sounds: a zero implies you measured and found nothing, and that is a different claim from "no data".

the practical habit: look at the taxonomy weekly, take the top type, fix it, watch it drop.

re-ask and abandon are your quality signal

these two are the closest thing to a quality metric you can get without a model, and they cost nothing extra because they fall out of turn structure.

re-ask rate is consecutive user turns with no assistant turn between them. the user said something, then said something else, without being answered in between. mechanically that is a failed response. it is a proxy, not a verdict, because sometimes people just add a thought, but it moves the right way when your agent gets worse.

abandon rate is conversations that end on a user turn. they asked, and either nothing came back or what came back was not worth replying to. also a proxy, also directionally honest.

what makes both worth trusting is that they are structural. no model interprets anything, so the same events produce the same number every time, which is what you want from something you will watch on a graph for months.

time to first token is the felt experience

total latency is the engineering number. time to first token is the number the user actually feels, because it is the gap between hitting enter and seeing evidence that something is alive. beyond a few seconds of nothing, people assume it broke and retry, which shows up as re-asks and inflates your load.

track it at p50, p90 and p99 like any other latency. p99 TTFT and abandon rate usually move together, which gives you a straight line from a technical number to a behavioural one.

resolution rate can only be measured if your app says so

this is the metric everyone wants and the one most tools quietly fake.

there is no honest way to compute, from turn events alone, whether the user's problem got solved. you can guess. a model can read the conversation and form an opinion. what you cannot do is measure it, unless the application itself knows: a ticket got closed, an order got placed, the user clicked "that solved it".

so resolution rate is null until your app sends a resolved bool on a turn. not zero, not estimated, null. once you send it, it is computed and real. that is a deliberate design choice: a missing number is annoying, an invented one is worse, because it will be quoted in a meeting six months later by someone who has no idea where it came from.

the order to add this in

1. send agent_tool_call events first. name, errored, error type, latency, client. this alone gives you three of the six numbers and it is the smallest change.
2. watch p90 and the error taxonomy for a week. you will find something. almost everyone does, usually one dominant error type or one tool with a bad tail.
3. add agent_turn events. conversation id, role, timestamp, and ttft_ms on the assistant turn. now you have re-ask, abandon, and TTFT.
4. decide what resolution means in your product, and send resolved when you know it. if you genuinely cannot tell, leave it null and say so out loud.
5. add text to turns only if you want labels. it is what makes conversation sampling possible, and it is your call whether to store it.
6. label a sample weekly with your own model for intent and sentiment, using the loop described here. free and unmetered, because it runs on your model, not ours.
7. keep the two kinds of number in separate columns forever. computed metrics get a trend line. inferred labels get a model name next to them and a labeled/unlabeled count.

if your agent has a front end, all of this sits in the same install as your web and product analytics, one MIT go binary, one snippet, one MCP connection that operates every project. the features list has the rest, /for/ai-apps is the short version for people building on models, and pricing is $19/mo for 2M events with a 14-day full trial and no card, or free forever self-hosted.

the summary: measure the seven computed things, be honest that resolution needs your help, and never let an inference sit in the same column as a measurement.