Knowing what your AI agent did, and what it cost you

Agent systems have a failure mode that traditional observability does not catch. Nothing crashes. No error rate climbs. Latency looks normal. Then the invoice arrives, and it is wrong. When you give a language model a loop and a set of tools, cost stops being a line item and becomes a behavior. A bug that would have been a stack trace in a normal service shows up in an agent system as a quiet spending multiplier that runs for days.

I run cost observability for an autonomous engineering agent in daily production use. It works across Slack, Sentry, and GitHub, and it spends real money every time it thinks. These notes are what survived contact with production.

Store raw events, not costs

The most important decision in the whole system is also the least glamorous: the telemetry table stores raw token counts, never computed dollar amounts.

The naive version writes a cost_usd column at request time. It works until the first provider price change. After that you have three bad options: migrate the table, backfill the history, or accept that every report mixing old and new rows is quietly wrong. Model prices change often, new models appear monthly, and cached-token pricing has its own rates. A cost column turns every one of those events into a data-integrity problem.

Storing raw counts makes pricing someone else’s problem at write time:

create table llm_usage (
  id bigint generated always as identity primary key,
  occurred_at timestamptz not null,
  model text not null,
  input_tokens int not null,
  output_tokens int not null,
  cache_read_tokens int not null default 0,
  source text not null,      -- which feature or agent made the call
  context_id text,           -- thread, task, or request it belonged to
  metadata jsonb not null default '{}'
);
-- no cost column, on purpose

Dollars are computed at read time from a model-to-price table. When a provider reprices, you update the table; the raw token history never changes, so there is no migration and no backfill. The token counts stay the ground truth, and the dollar figures are what they honestly are: estimates derived from the current table. If you need historically exact dollars, the same raw data supports a pricing table with effective dates; the point is that the decision to store tokens instead of money is what keeps both options open.

Attribution beats totals

A total spend number is almost useless on its own. The question that matters during an incident is never “how much are we spending.” It is “which part of the system started spending differently, and why.”

That is what the source and context_id columns are for. Every LLM call carries a label for the feature that made it and an identifier for the piece of work it belonged to. The discipline costs one extra argument at every call site and pays for itself the first time spend jumps. An alert that says spend is up threefold is noise. An alert that says spend is up threefold and most of the increase is coming from one source is the start of a diagnosis.

We went one step further and attached investigation hints to each source: a short mapping from “this source is spiking” to the configuration most likely responsible, things like retry limits, context-window settings, or a model override. The alert names the suspect. Whoever is on call starts from a hypothesis instead of a blank dashboard.

Alert on change, and let people sleep

Cost anomalies are bursty. A misconfigured loop does not spike once; it spikes every minute until someone stops it. If your alerting fires on every evaluation, the person on call gets fifty pages that all say the same thing, and by the third incident they have muted the channel.

Two rules fixed this for us:

1. Alert on deltas against a baseline, not on absolute thresholds. Absolute numbers rot as usage grows; a threshold that was alarming in March is a normal Tuesday by August.
2. Put a cooldown behind every alert. We keep cooldown state in Redis: when an anomaly fires for a source, that source goes quiet for a set window no matter what the numbers do. One spike, one page.

The checklist

If you are building an agent system and want the money side to be boring, this is the shape of it:

1. Store raw token counts per call. Compute money at read time from a pricing table with effective dates.
2. Tag every call with a source and a context id. Make the tag mandatory at the call site.
3. Alert on change relative to baseline, with per-source cooldowns.
4. Put the likely cause in the alert itself.
5. Keep the ledger queryable by the people who pay the bill, not just the people who wrote the agent.

None of this is glamorous. It is bookkeeping. But an agent only gets to keep its autonomy while somebody can say exactly what it did and what that cost, and the ledger is what makes that sentence possible. The best outcome is a boring one: the invoice arrives, and it matches what you already knew.