Your Agent's Token Bill Isn't the Model's Fault. It's the Context You Resend Every Turn.

The line item nobody budgets for

When a business first prices out an AI agent, the conversation is almost always about which model to use and what it costs per call. That's the wrong unit of analysis. An agent doesn't make one call — it makes a chain of them, and on most architectures, every call in that chain resends the same system prompt, the same tool definitions, and often the same retrieved documents, all over again. If your agent takes six steps to resolve a support ticket or process an invoice, you are not paying for one prompt. You're paying for six copies of your system prompt, with a few hundred tokens of actual new content tacked onto each one.

This is the part of agent economics that doesn't show up in a demo, because a demo is one interaction. It shows up in month two of production, when the invoice reflects thousands of ticket resolutions times six redundant calls each, and someone asks why the per-resolution cost is three times what the pricing page implied.

Why re-sending context costs real money

LLM APIs bill by the token, and until recently, a token sent for the tenth time cost exactly what it cost the first time. An agent's system prompt (instructions, tool schemas, few-shot examples, retrieved context) is frequently the bulk of what gets sent on every turn — the actual new input (the user's latest message, a tool result) is often a small fraction of the request. For a multi-step agent, that static prefix gets paid for again and again, turn after turn, with no discount, unless something intervenes.

That something is prompt caching, and it's no longer optional infrastructure for anyone running agents at real volume — it's the difference between a cost model that scales and one that doesn't.

What caching actually buys you

Anthropic's prompt caching lets you mark a prefix of a request — system prompt, tool definitions, a long document — as cacheable. The first call that includes it pays a write premium: 1.25x the base input price for a 5-minute cache, or 2x for a 1-hour cache. Every subsequent call that hits the same cached prefix within that window pays roughly 0.1x the base input price for those tokens — a 90% discount on the part of the prompt that didn't change. OpenAI's equivalent is automatic rather than something you configure: prompts over 1,024 tokens get checked against recently seen input, and matching tokens are billed at a 50% discount, with no code change required — you just read cached_tokens back in the usage data to confirm it fired.

Run the arithmetic on a real agent: if 80% of your input tokens are the same static context on every step, and you're making five steps per task, caching isn't a marginal optimization — it's the difference between paying for one system prompt and paying for five.

The gotcha that catches agent builders specifically

Caching has a minimum prefix length to kick in, and that minimum isn't fixed — it varies by model and model generation, and providers have changed it between releases without much fanfare. The failure mode that matters is this: a prefix shorter than the minimum doesn't throw an error. It just silently doesn't cache. You'll see cache_creation_input_tokens: 0 in the response and nothing else flags it. A team that assumes caching "just works" because they set the right parameter can run in production for months paying full price on every call and never notice, because nothing broke — the agent still works, it just costs more than it should.

The second, more structural trap is ordering. Caching matches on an exact prefix. If your static content (system prompt, tool schemas) comes first and the variable content (conversation history, the user's latest message) comes last, the cache hits cleanly on every call. If a developer — often without realizing the cost implication — puts anything variable before the static block, or interleaves retrieved context with per-turn state, the cache invalidates every single time, and you're paying the write premium repeatedly instead of the read discount. This is an architecture decision, not a toggle, and it's easy to get backwards when the static-vs-dynamic split isn't obvious at the code level.

Where this intersects with RAG

If your agent pulls fresh documents from a retrieval step on every turn, those documents are — by definition — not the same prefix twice, so provider-level prompt caching won't help the retrieved content even if it helps the system prompt and tool definitions around it. That's a different problem, usually solved (if at all) with a separate semantic caching layer that checks whether a similar query was answered recently, rather than relying on exact-prefix matching. Don't let a vendor claim "we use prompt caching" as if it automatically covers the RAG-heavy part of the pipeline — ask specifically what's cached and what isn't.

What to ask if you're buying an agent

If you're evaluating a vendor or an internal team building an agent that will run at volume, three questions separate people who've engineered for cost from people who haven't tried yet:

• Is the prompt structured with static content first, dynamic content last? If they don't know what this question means, caching isn't actually working for them.
• Do they monitor cache hit rate in production, not just assume it from having "turned it on"? A silent miss looks identical to a hit in every way except the bill.
• For RAG-backed agents, what happens to the retrieved-document portion of the cost — is it cached at all, and if not, what's the plan?

The takeaway

Prompt caching can cut 60-90% off the repeated-context portion of an agent's token bill, but only if the static and dynamic parts of the prompt are actually separated and the cacheable prefix clears the model's minimum length — and neither of those is something you get by default. If you're quoting or budgeting for an agentic system, ask to see the token breakdown per step, not just the model's list price; that's where the real cost either disappears or quietly triples.