Managing LLM Token Costs Like Cloud Costs
■ Problem: LLM token costs show up only as a monthly total per API key, so the bill grows without anyone knowing which feature is spending what.
■ Solution: Apply what you already do for cloud costs, allocation, pricing choices and unit cost tracking, to tokens.
■ Result: AI spend that used to be one line at month-end becomes cost per request by feature, and the calls you can move to batch or cache become visible.
Two months after a call summary feature went live, finance showed up with the LLM API bill. All the engineers had was a monthly total attached to a single API key. There was no way to tell how much belonged to the summary feature, the internal search chatbot, or the test scripts borrowing the same key.
Can LLM token costs be managed like cloud costs? Short answer: yes. The tools are already built into the pricing.
LLM tokens don't have a single price
Like the cloud, it's pay-as-you-go. The difference is that even within one model, different kinds of tokens cost different amounts. The FinOps Foundation's token economics paper puts it this way: "Most providers charge more for output tokens than input tokens, sometimes by a factor of three to five." The longer the answers a feature asks for, the faster the bill grows.
The same paper also says, "The token is the billing unit, not the value unit." A rising token count on its own doesn't tell you whether the money was well spent.
The levers you used for cloud exist for tokens too
Look at the pricing structure and a familiar shape appears.
- Send calls that don't need an immediate answer to batch. Both Anthropic's documentation and the Amazon Bedrock pricing page list batch processing at 50% of the standard rate (for some models on Bedrock)
- Cache long system prompts that repeat. Per Anthropic's documentation, cache reads cost 0.1x the base input price (with some model exceptions)
- If usage is steady, a throughput commitment is an option. Bedrock's Provisioned Throughput is priced per model unit per hour, so on-demand vs. commitment works much like it does for instances
Caching isn't free. In the same documentation, a 5-minute cache write costs 1.25x the base input price, so caching prompts that are rarely reused actually costs you more.
Before and after: what changes
Before, finance gets the bill total at month-end, and engineers guess at causes from a total tied to one key. Summaries and the chatbot all run at real-time rates.
After, each feature gets its own API key, and calls are logged with the feature name. The same person checks "cost per call summary" every week. The overnight bulk summaries of call history move to batch. How to assign an owner at the moment a cost is created is covered in our post on AI agent cost allocation.
Back to finance's question. The answer is now cost per request by feature and the share of calls moved to batch. Turning those numbers into a budget picks up in our post on AI cost budgets.
▶ Talk to OpsNow about your cloud costs
Frequently Asked Questions
Q. Can LLM token costs be split like tags?
Yes. Features equivalent to cloud resource tags vary by provider, but separating API keys by feature or logging the feature name with each call has the same effect.
Q. When should we use batch processing?
For work that doesn't need results right away. Bulk summaries of call history, document classification and high-volume evaluation calls are typical.
Q. Is prompt caching always worth it?
No. Cache writes cost more than the base input price, so it only pays off for prompts that reuse the same opening many times.
