The Missing Span: Putting Token Cost Inside Your Observability Trace

By Author: Robert
Total Articles: 30
Comment this article

How token spend became the newest pillar of AI observability, and what a month of Datadog observability signals tells us

For most of the last two years, the story teams told themselves about AI cost was a finance story. Someone pulled a usage report at month-end, squinted at a number bigger than expected, and asked: where did all of this go? The answer was usually a shrug. The bill was real, but the reasoning behind it stayed invisible, which is strange, because closing that kind of gap is what mature Datadog observability already does for the rest of the stack.

That shrug is now the single most interesting problem in AI observability. A line item that used to sit quietly on a finance report is becoming a signal any serious enterprise observability practice is expected to trace and act on. The industry has a name for it now: tokenomics!

Why Counting Was Never Going To Be Enough

Look at what production AI became while everyone argued about which model was smartest. Datadog's State of AI Engineering research, drawn from thousands of organizations running AI in production, reached a conclusion that should ...
... reframe how you think about cost: the primary barrier to reliable AI at scale is not model intelligence, it is operational complexity.

Nearly seven in ten companies now run three or more models alongside tangled agent workflows. Around five percent of AI model requests fail in production, and close to sixty percent of those failures trace back to capacity limits. Average tokens per request more than doubled for typical teams and quadrupled for the heaviest users. This is the terrain that modern Datadog observability was rebuilt to cover.

Sit with that shape. More models, more handoffs, more tokens per call, and a slice of it silently failing. A monthly spend report cannot tell you which moving part is burning money; only a connected Datadog observability trace can. This is where Datadog AI observability earns its keep, because counting tokens will never reveal that a retrieval step is looping, that an agent keeps re-reading the same file, or that a tool nobody uses is quietly taxing every request. That is a job for real AI observability, not a billing export.

The Tokens You Spend Before An Agent Acts

A large share of token cost is incurred before an agent does a single useful thing. Recent analysis suggests one capability can occupy roughly ten thousand context tokens through a conventional MCP server, versus roughly one hundred tokens as a compact skill. Many production setups carry tens of thousands of tokens of tool definitions before a conversation even begins.

Call it the version of paying rent on an office nobody visits. The capability sits in context, costing money on every call, whether or not the agent reaches for it, and Datadog observability is where that idle cost becomes visible. You will never catch it on a bill.

You only see it in a trace, which is why teams standardizing on Datadog observability now treat context bloat as a monitored quantity rather than a hidden tax. Some of that consumption is not even accidental: poorly behaved tools can push agents into cyclic loops, and analysts warn that deliberately inflated consumption is becoming a security concern. Once cost can be weaponized, observability has to answer for dollars as much as for performance.

From Counting to Tracing to Yield

The clearest sign something has matured is when it grows a vocabulary, and this space just did. People talk about token yield, the useful work produced relative to tokens consumed, and separate it from token maxing, the habit of treating raw consumption as a proxy for productivity. The through line is the same shift Datadog observability went through with infrastructure cost: measure the spend, attribute it, then optimize it.

Attribution is the hinge. Cloud cost management only became powerful once a line item could be tied to a team, a service, and a deploy. Agent cost walks the same path, which is why Datadog AI observability now races to attach a dollar figure to every span. Cost intelligence does exactly that, tying spend back to the specific agents and workflows generating it, a move from raw usage toward business-level accountability. This is AI observability doing what it has always done, now pointed at spend.

What It Means If You Run AI In Production

If you treat cost as a finance function you reconcile after the fact, you are structurally too late, and Datadog observability exists precisely to close that lag. By the time the bill lands, the looping agent has already looped, and the over-provisioned model has already run for a month.

Token cost belongs with latency, errors, and traces, because it comes from the same source and is fixed with the same instrument. That is the whole premise of AI observability. The teams pulling ahead can open one trace and follow the full chain from prompt through retrieval, model inference, and tool calls, with cost attached at every hop. That is a properly instrumented observability view doing what it was designed to do.

The good news is that this is buildable today. A well-configured Datadog observability practice already captures the request chain cost attribution depends on, so adding token spend is really a new lens on data you already collect. Counting tokens was never the point. Understanding them is, and that is fast becoming the baseline for enterprise observability.

At Crest Data, our Agent Observability practice helps teams instrument AI applications with Datadog AI observability so token cost, model quality, and reliability live in one correlated view. When a Datadog observability rollout scales faster than your grasp of the AI bill behind it, closing that gap is exactly what this discipline is built to do.

For more information please visit https://www.crestdata.ai/solutions/datadog/