At VMware Explore in Las Vegas recently, I sat in on a session about tokenomics. In the session, J.R. Storment, executive director of the FinOps Foundation, put a number on something a lot of us had been circling. Companies are estimating that only about a quarter of their total AI cost is direct model consumption, that is, the tokens. The other three-quarters is GPU and supporting infrastructure: computing, memory, database, and the services and labor wrapped around all of it. (Full disclosure: We were one of the cosponsoring organizations at the event.)
If you're managing AI spending based on your token bill, you may only be addressing a quarter of the problem.
The ratio isn't fixed—it's a function of your architecture
What portion of total enterprise AI costs actually comes from token usage? The number that matters for your organization depends on what you've actually built, and the range is wide.
At one end, you may have a thin wrapper around a model API. The user types, you forward it, and you display what comes back. At this end, it really is mostly token cost. Your infrastructure overhead is a web server. If that's your system, the 25% figure will feel wrong to you, and you'll be right.
At the other end is anything doing retrieval, orchestration, or multi-step work. In these cases, the large language model (LLM) call is one operation inside a much larger pipeline, and the pipeline is where the money goes.
The uncomfortable part is that this second architecture is what almost everyone is migrating toward. The ratio isn't a static fact about AI. It's a description of what happens to your cost structure as your AI application grows up.
What's actually happening around the LLM call
Why are LLM model token bills only a fraction of total AI infrastructure spending? When you type a prompt into a chatbot, there's work happening on both sides of that model call that you never see. Look at the anatomy of a single request in Clarity Vaia.
Before anything reaches the model, there's prompt collection, configuration of what follow-up questions might be needed, and logic that determines which data to retrieve and point the model at. That retrieval step is doing real work against real infrastructure, including database reads, memory consumption, and computing usage. Plus, this work happens on every request, whether or not the model ends up needing all of it.
On the way back, the result has to be packaged before it's usable. Critically, that packaged result should be reused rather than regenerated. Every time you send an equivalent request back through the model instead of serving a cached result, you pay the token cost and the surrounding infrastructure cost again, all that for an answer you already had.
None of that is LLM cost. It's infrastructure cost incurred entirely in service of AI, and it doesn't appear anywhere on a token bill.
Caching is the lever nobody's pulling hard enough
How can prompt caching reduce AI retrieval pipeline and computing expenses? In a retrieval-heavy system, a meaningful share of requests are semantically close enough to something already answered. As a result, regenerating these requests is pure waste. Plus, the waste is doubled, because you're re-running the retrieval pipeline as well as the model call.
Most teams treat caching as a latency optimization. In an AI workload, it's a cost architecture decision, and it's the one place where an engineering choice shows up directly in the finance conversation. This is exactly why the two disciplines can't stay separate anymore.
Why nobody tracked this before, and why that breaks now
Historically, none of the surrounding infrastructure got traced to the workload that caused it. It was absorbed into general overhead and allocated in a top-down fashion, whether a headcount split or something equally coarse. Nobody bothered to say, "This database spending exists because of this specific application." That was a defensible shortcut, because the infrastructure attributable to any one application was small enough that misallocating it didn't change any decision.
That's the assumption that just broke. When AI-driven processing becomes a large share of what an application costs to run—relative to the people maintaining it—the coarse allocation stops being a rounding error and starts being the thing you're trying to measure. You can't optimize a workload when you've spread its costs evenly across a department.
And the failure isn't just under-counting. It's mis-ranking. Two AI features with identical token spending can have wildly different true costs depending on how much retrieval and orchestration each one drags behind it. If you're comparing them solely on tokens, you'll confidently pick the wrong one to scale.
This is becoming a discipline, not a spreadsheet
Storment's session wasn't an isolated observation. At FinOps X in June, he announced the launch of the Tokenomics Foundation under the Linux Foundation. Working alongside the FinOps Foundation, the organization is a vendor-neutral body developing standards and benchmarks for AI cost governance. (See a prior post to find out why Broadcom joined the Tokenomics Foundation as a founding member.)
That matters for how you should read vendor posts, including ours. Tokenomics is not just a vendor term, it is becoming an industry discipline with open standards attached. We named our product ValueOps AI Tokenomics because we think that discipline is the right frame. The standards work happening at the Linux Foundation is a good thing for buyers, because it makes it possible to compare vendors based on something other than their respective marketing claims.
Summary
The practical takeaway is about architecture, not procurement. Whatever tooling you use, the requirement is the same: Trace the full stack a given AI workload runs on, not just the model calls it happens to make. Otherwise, you're pricing a quarter of the bill and calling it the whole thing.
This is why ValueOps AI Tokenomics doesn't stop at token consumption. For workloads on private infrastructure, the solution ties token consumption back into the same resource data already tracking computing, storage, and the rest of the stack. As a result, you see what an AI workload actually costs, not just what the model billed you for.
Frequently asked questions
Q: Why are token bills an insufficient metric for evaluating overall AI expenses?
A: Token bills capture direct model interaction, which typically represents only 25% of an AI application's total cost. The remaining 75% goes toward supporting infrastructure, such as computing, memory, database reads, and orchestration pipelines.
Q: How does application architecture affect total AI operational spending?
A: Thin API wrappers cost very little beyond model tokens because their infrastructure overhead is minimal. In contrast, applications using retrieval, orchestration, or multi-step pipelines consume significant database, memory, and computing resources. This resource consumption can occur before and after the LLM call occurs.
Q: What role does semantic caching play in AI costs?
A: Caching serves as a major cost-reduction lever by reusing existing responses for semantically similar prompts. Reusing answers prevents duplicate token charges while eliminating redundant, expensive database retrieval and pipeline processing steps.
Q: What is the Tokenomics Foundation and why was it established?
A: Launched under the Linux Foundation alongside the FinOps Foundation, the Tokenomics Foundation develops open, vendor-neutral standards and benchmarks for AI cost governance. Through these standards, the foundation helps leaders gain a basis for objectively comparing the merits of vendor solutions.
