FinOps for LLM Spend: Three Reasons Your Bill Changed

TL;DR

When your LLM bill moves month to month, one of three separate things caused it. Your usage changed (more or fewer tokens). Your pricing changed (a discount kicked in, a markup went away, cache-hit rates got billed differently). Or the way your app talks to the model changed (routing, caching, model choice). A dashboard that only shows "spend down 40%" without splitting those three apart is how teams end up thanking the wrong lever, and repeating the wrong fix next quarter. Here is a simple framework for pulling them apart before you claim you saved money.

Why this matters

Cloud FinOps figured this out a decade ago. Everyone in the AWS world knows that a lower bill can come from three unrelated things. Fewer instance hours. A Savings Plan kicking in. A rightsizing project. Confusing them is bad practice, because each one points to a different next step.

LLM spend is going through the same learning curve, from scratch, with teams that were not around for the cloud version. And it is harder here, because more moving parts sit between "user typed a prompt" and "line item on the invoice." Model choice, cache hit rate, provider markup, batch discounts, per-org rate limits, and gateway routing all pull on the number.

The framework below names the three axes. In practice, most real cost changes are a mix of movement on some or all of them. The trap is treating a combined change as if one lever caused it.

Axis 1: Usage change

This is what most dashboards actually show. It is the change in how much work your app is asking the LLM to do, before any pricing or architecture decisions.

What causes it

More or fewer end users
More or fewer requests per user
Longer or shorter prompts (context growth, bigger RAG documents)
Longer or shorter completions (chattier or terser system prompts, JSON-mode differences)
New features that add or remove LLM calls

How to measure it on its own

Hold everything else fixed. Compare input_tokens + output_tokens month to month, not dollars. If your token count changed but your cost per token did not, the whole delta is usage.

Common mistake

Crediting a lower bill to "we optimized" when the real story is that fewer users showed up. That is a churn signal, not an efficiency signal. Congratulations, you saved money on the LLM bill and lost a third of your customers.

The FinOps question worth asking every month

What would this bill have been at last month's cost per token? If the answer is close to this month, usage moved. If it is far off, usage is flat and something else is doing the work.

Axis 2: Pricing change

Pricing change is what changed about how you buy the tokens, not what tokens you bought. Most engineering teams never see this axis because it lives in contracts, procurement, and BD conversations.

What causes it

A committed-use discount kicked in (OpenAI, Anthropic, and Google all offer volume tiers)
Provisioned throughput on Bedrock replaced on-demand rates
A gateway or aggregator markup got added or removed (5 to 10 percent on OpenRouter, 20 to 40 percent on some enterprise wrappers, zero on flat-fee BYOK gateways)
Cache-hit rates got billed differently (Anthropic charges 10 percent of the base for cached input, OpenAI dropped cached-input rates in mid-2026, DeepSeek moved to a peak and off-peak model on 2026-08-16)
Batch API discounts (50 percent off if you can wait)
Switching between BYOK and managed-invoice paths through the same infrastructure

How to measure it on its own

Compute the effective dollars per million input tokens and dollars per million output tokens for the same model, at the same request pattern, on last month and this month. If usage is flat and the effective rate changed, the whole delta is pricing.

Common mistake

Giving engineering credit for a bill drop that was really the finance team negotiating a committed-use tier. The engineers who worked on caching feel like the cache saved money. The finance team knows the commit saved money. Both are right, and both are underselling. Without splitting them, one team's story becomes the official story, and the real driver gets missed.

The FinOps question worth asking every month

If usage had been identical to last month, what would we have paid at this month's contract terms? That number is your pricing-only impact.

Axis 3: Architecture change

Architecture change is what the gateway, routing, and caching layer changed in how the workload got served, before it ever reached the model.

What causes it

Cache hit rate moved (prompt cache, semantic cache, RAG retrieval cache)
Cheaper-model routing (a prompt that used to hit Sonnet 5 now hits DeepSeek V4 Flash on the same-tier arbitrage rule)
Model choice changed (dropping GPT-5.6 Sol for GPT-5.6 Terra on a workload that did not need premium reasoning)
Retry policy tightened (fewer retries after upstream 5xx errors)
Failover happened (primary provider was down, secondary provider costs different)
Batch consolidation (multiple small requests merged into one)
Guardrail overhead (prompt-injection detection adds one moderation API call per request)
MCP tool-use loops resolved in one round trip instead of three

How to measure it on its own

For a fixed workload and fixed contract terms, compare the effective cost per request. If the token bill dropped but the request count and unit rates are flat, the architecture in front of the model changed. Your gateway logs are the source of truth here. You need per-request records of model chosen, cache hit or miss, retry count, and failover events.

Common mistake

Treating a cache-hit-rate improvement as "we saved money" when the customer's prompt pattern happened to shift toward more repetitive requests. That is usage pretending to be architecture. Real architectural wins move the hit rate on a stable prompt pattern.

The FinOps question worth asking every month

If we had served this month's workload with last month's routing rules, cache configuration, and model selection, what would we have paid at this month's contract terms? That number is your architecture-only impact.

A concrete example

Two teams. Both send 100 million Sonnet 5 tokens per month.

Team A

On-demand direct to Anthropic
Retail pricing: $2 per 1M input, $10 per 1M output
5-to-1 input-to-output ratio, so blended cost is $20 per 6M tokens
100M tokens per month works out to about $2,000 per month

Team B

Has a 10 percent committed-use discount with Anthropic
Routes through a flat-fee BYOK gateway ($25 per month)
Prompt cache hit rate is 30 percent
Same 5-to-1 ratio, same 100M tokens

Team B math:

Full-price tokens: 70 percent of ($333 input + $1,667 output) minus 10 percent commit = about $1,260
Cache-hit tokens (30 percent, at 10 percent of the input rate): 30 percent of $333 times 0.1 = about $10
Gateway fee: $25
Total: about $1,295 per month

Same model. Same workload. 35 percent unit cost difference. Now the questions.

How much of the gap is usage? Zero. Both teams sent identical traffic.
How much is pricing? About $200. The 10 percent commit on 90 percent of the traffic.
How much is architecture? About $500. The 30 percent cache hit rate and the routing.
How much is the "gateway is expensive" tax? Negative $475. The $25 flat fee is tiny compared to the savings the routing enables.

If Team B's dashboard just says "we spend 35 percent less on LLMs than Team A does," they cannot answer any of those four questions. If Team A ever tries to close the gap, they will guess wrong about which lever to pull.

The discipline

Every monthly LLM cost review should answer the same three questions, in this order.

What did usage do? Token count month to month. Request count month to month.
What did pricing do? Effective dollars per token, by model, month to month.
What did architecture do? Effective cost per request, at fixed usage and fixed contract terms, month to month.

The three numbers should add up to the total dollar change. If they do not, you have unattributed variance. That usually means a fourth thing is quietly moving. Someone changed the default model. A new feature launched that no one told finance about. A retry storm during a provider outage bloated the bill.

Teams that do this monthly develop something the teams that skip it lack. An accurate story about why the LLM bill did what it did. Which is what makes future budgets defensible and future optimizations credible.

What this means for gateway choice

This post is on a gateway vendor's blog, so it is worth being honest about where the framework lands on that question.

How you route affects the architecture axis heavily, and the pricing axis in some paths. A flat-fee BYOK gateway is really two levers bundled together. It caps the "we take a percent of your tokens" markup at zero (pricing) and lets you route across model choice, cache, and failover (architecture). Whether that combination beats a direct-to-provider setup depends entirely on your cache hit rate, your model diversity, and whether you already have committed-use pricing negotiated.

But that decision has to be made with the numbers split apart. Not with a "we saved money" claim on either side. The gateway is a lever. Whether pulling it is worth it is a FinOps question, and it needs the three-axes framework to answer honestly.

The bigger point

Cloud FinOps became a discipline because too many teams confused usage growth with rising unit cost, or Savings Plan kicking in with a rightsizing win. The result was that when the AWS bill spiked, nobody could tell whether it was a scaling problem, a pricing problem, or a wasted-resource problem. And the fix was always a guess.

LLM spend is going through the same curve. Faster. With fewer reflexes in place. Teams that build the reflex early (split before you attribute, name the axis before you name the win) will make better product bets in year two of their AI investment. Teams that skip it will keep pulling levers and never quite know which one moved the number.

If you are building the LLM cost story at your company, the three axes are the first tool. The dashboards, the gateway choice, the vendor contracts all follow from having the vocabulary to talk about which axis moved.

Sources: