Every team building on top of large language models eventually has the same moment: the invoice lands, and it’s higher than anyone expected. The instinctive reaction is to blame the pricing page, surely the rates changed, surely there’s a hidden fee. But pull the number apart and the pricing was never the mystery. What’s missing is a basic audit of where the tokens are actually going.
Most teams can tell you what model they’re using. Very few can tell you what’s inside the request that model receives every single time it’s called. That gap, between “we use Claude” and “here’s exactly what we send Claude on every request”, is where budgets quietly get away from people.
This piece walks through that gap in three parts: what a token actually is, what tends to drive usage up without anyone noticing, and the handful of levers, caching chief among them, that bring the number back down without making the product worse.
What tokens actually are
Before a model reads a single word of your prompt, a tokeniser has already chopped it into smaller pieces called tokens, subword units that sit somewhere between individual letters and whole words. “Understanding” might become one token or split into “under” and “standing” depending on the tokeniser’s vocabulary. The model never sees your sentence, it sees this chopped-up version.
Here’s the part that trips people up: the same sentence can produce a different token count on different models, because each provider trains its own tokeniser on its own vocabulary. A prompt that’s cheap on one model isn’t guaranteed to be cheap on another, and rough word-count estimates won’t tell you much across providers.
Why does any of this matter outside of academic curiosity? Because tokens aren’t just a linguistic detail, they’re the literal unit both your bill and your context window are measured in. Every optimisation decision downstream of this guide comes back to that one fact.
Why this deserves its own analysis
It’s tempting to treat tokens as a rounding error, just part of “using AI.” Three things make that a mistake:
Cost. Input and output tokens are priced separately, and at any real scale, the difference compounds fast. A prompt that’s 10 percent bloated across a million calls a month isn’t a rounding error anymore.
Context limits. Every model has a hard ceiling on how many tokens it can hold in a single request. Cross it, and content gets truncated, sometimes silently, sometimes with an error, always with a debugging headache attached.
Latency. Output tokens are generated one at a time, in sequence. There’s no shortcut. A response that’s twice as long takes roughly twice as long to finish streaming back, which means token bloat isn’t just a cost problem, it’s a user-experience problem too.
Where the tokens actually go
This is the audit most teams skip. Pull apart any real production request and it’s rarely just “the user’s question.” It’s usually a stack of four things:
System prompts. These get resent on every single call. A bloated system prompt doesn’t cost you once, it costs you on every request, forever, until someone trims it.
Conversation history. In a multi-turn chat, most implementations resend the entire prior conversation with every new message, one of the quietest and biggest cost drivers in production LLM apps. A ten-turn conversation isn’t costing you for turn ten, it’s costing you for all ten turns, again, every time.
Retrieved context and RAG documents. Whatever gets pulled in from a vector database or search index gets injected wholesale into the prompt, often as much larger chunks than the answer actually required.
Tool definitions and results. Easy to forget, but the schemas describing available tools, and the raw data those tools return, count against the exact same budget as everything else.
If you’re building a mental model of this, picture a pie chart: system prompt, conversation history, retrieved documents, and output, all competing for the same slice of spend. For most chat-style products, history and retrieved context dominate, not the clever prompt engineering everyone spends their time on.
Measuring it properly
You don’t need to guess at any of this. Most SDKs return an exact usage object on every response, precise input and output counts, the ground truth for whatever you’re trying to audit.
One catch: if you’re estimating token counts before sending a request, for budgeting, chunking, or pre-flight checks, use the target model’s own tokeniser. A generic or mismatched tokeniser will systematically miscount tokens for a different model’s vocabulary, throwing off any estimate built on top of it.
The habit that actually pays off, though, is logging usage per component, not just per request. A single “tokens used: 4,200” line tells you nothing. Knowing that 2,800 of those came from resent conversation history tells you exactly where to spend your next hour of optimisation.
The single biggest lever: prompt caching
If there’s one change worth making before any other, it’s this one.
Anthropic’s prompt caching works by storing a processed representation of the static parts of your prompt, system instructions, reference documents, tool definitions, so the model doesn’t have to recompute them from scratch on every call. The next request that reuses that same static content reads it from cache instead of processing it fresh.
The savings aren’t marginal. Cache reads are priced at a steep discount against standard input tokens, and the real-world numbers back that up: one company cut total LLM spend by 59–70 percent simply by raising their cache hit rate from 7-percent to 84-percent. That’s not a clever prompt trick, that’s a structural fix that pays out on every single call afterward.
The practical rule of thumb is simple to state and easy to violate: put static content first, dynamic content last. System prompt, instructions, reference documents, all up front, in a stable, unchanging order. User-specific or per-request content goes at the end. Get that ordering wrong, even slightly, and caching quietly stops working, you keep paying full price and have no idea, because nothing throws an error. It fails silently, which is exactly what makes it worth checking rather than assuming.
Other optimisation levers worth knowing
Caching does the heavy lifting, but a few other habits compound nicely alongside it:
- Trim conversation history instead of resending it whole. Summarise older turns rather than carrying the full verbatim transcript forward on every message.
- Right-size the model. Reserve your largest, most expensive model for tasks that genuinely need that level of reasoning, and route simpler tasks to something smaller.
- Constrain output format. Asking for structured output, JSON, a fixed schema, cuts down on the rambling, unstructured prose that eats output tokens for no real benefit.
- Chunk RAG retrieval more precisely. Pull the paragraph that answers the question, not the whole document it lives in.
Common mistakes teams make
A few patterns show up again and again in these audits:
- Assuming token counts are portable across models, when tokenisers differ enough to throw off any cross-model estimate.
- Trying to cache content that changes on every request, which earns you nothing but the write premium, with no read-side discount to offset it.
- Spending hours polishing prompt wording while the actual cost driver, unmanaged, endlessly resent conversation history, sits untouched.
The takeaway
Token consumption isn’t a wordsmithing problem. It’s a structural, engineering one, and most of the waste in production systems comes from repetition, not from how the prompt is phrased. Clever phrasing shaves a few tokens off a single request. Fixing how you handle conversation history or caching changes the shape of your bill permanently.
Don’t try to fix everything this week. Pick one component, conversation history is usually the biggest offender, and actually measure what it’s costing you before you touch anything else. The audit is the part everyone skips, and it’s the part that tells you where to spend your effort.
