LLM Cost Optimization: Caching, Routing, and the Token Waste Hiding in Your Prompts

TL;DR

• Prompt caching is the highest-ROI single change for most production LLM systems: Anthropic bills cached reads at 10% of the standard input rate, and the break-even arrives at the second cache hit.
• Model tiering routes the majority of traffic to a cheaper model and reserves frontier models for tasks that actually need them — done with static rules first, learned routing later.
• The largest category of wasted tokens is usually not the system prompt — it is stale chat history, over-retrieved RAG chunks, and agent rework loops replayed at full price.
• A gateway (LiteLLM self-hosted or OpenRouter managed) gives you a single control point for caching, routing, fallback, and spend tracking.
• Measure prompt_tokens, completion_tokens, and cache_read_input_tokens per request before changing anything. Optimizing without a baseline is guesswork.

---

Why Did the Bill Go Up When Token Prices Went Down?

Token prices fell 80% between 2025 and 2026. Enterprise AI bills went up. That gap is not a pricing problem. It is a governance problem. Usage grows to fill whatever budget feels comfortable, and most of that growth is invisible until the invoice arrives.

The pattern is predictable. A team ships an AI feature. Other teams want the same. Agent count grows. Each agent sends the full system prompt on every call, retrieves more context than the task needs, and runs on a frontier model because no one has taken time to evaluate whether a smaller one would do. Most development teams squander 40–60% of their token budgets on suboptimal implementations. The fix is a stack of compounding moves applied in order: measure first, cache what repeats, right-size the model per task, then compress what remains.

Step One: Measure Before You Cut

Most teams optimize the wrong thing because they haven't measured where their tokens actually go. Log prompt_tokens, completion_tokens, and cache_read_input_tokens on every API response. If you are on Anthropic, the Usage & Cost API gives post-hoc breakdowns by model and cache tier.

Research shows that review and rework loops consume roughly 59% of tokens on average — not the initial generation. Input context growth, not prompt size, is usually the main cost driver. That finding inverts the usual instinct to tighten the system prompt first. The system prompt may be 500 tokens; the conversation history replayed on turn twelve of a failed agent loop may be 30,000.

Prompt Caching: The Highest-ROI Move

Prompt caching reduces costs and latency by reusing previously processed portions of your prompt across API calls. Instead of reprocessing the same large system prompt, document, or conversation history on every request, the API reads from cache at a fraction of the standard input price.

Tag the stable portion of your prompt with cache_control: { type: "ephemeral" }, pay 1.25x normal input price on the first request (5-minute TTL) or 2x (1-hour TTL), then 0.10x on every subsequent request within the cache TTL. The break-even is the second cache hit. After that, every call reads at a 90% discount.

Anthropic lets you place up to four cache_control markers anywhere in the request, meaning a retrieved document inserted mid-conversation can still cache alongside the system prompt and tools as a separate block. For RAG workloads, this is a real structural advantage.

Three situations where caching does not pay its way: short, mostly-variable requests where a 300-token classification prompt has nothing meaningful to cache; output-bound workloads where a 50-token prompt producing 5,000 tokens of output is unaffected by input caching; and very low volume, where the cache write cost is paid repeatedly before enough reads accumulate to break even.

Semantic caching sits one layer above provider-level caching. Semantic caching can cut API costs by up to 73%, while prompt optimization, context engineering, and RAG tuning provide additional savings. A semantic cache stores the embedding of a query and its response. When a new query arrives with high cosine similarity to a stored one, it returns the cached answer without touching the LLM. The failure mode is a threshold set too low, which returns stale answers confidently. Calibrate against real traffic, not a toy dataset.

Model Tiering: Routing the Easy Majority

A routing layer estimates each request's difficulty and dispatches accordingly — routine tasks to small low-cost models, hard reasoning to frontier models. The evidence says price spreads of 10x to 30x inside model families make workload-to-model matching worth doing, yet the largest and safest savings come from static tiering, prompt caching, reasoning-effort caps, and batch APIs before any learned router fires.

A practical escalation pattern: run a mid-tier or small model first; if a structured confidence check fails — the answer is low-confidence, schema-invalid, or flagged by a verifier — escalate to a frontier model. Track the escalation rate as a product KPI. A rising rate tells you the cheap model is being asked to do too much. If the escalation rate stays below 20%, the tiering is working.

Which Gateway Should You Use?

Gateway	Hosting	Best for	Main trade-off
LiteLLM	Self-hosted (Docker + Postgres + Redis)	Teams that need data residency and full config control	You operate the full stack; real engineering overhead
OpenRouter	Managed SaaS	Fast access to 100+ models with no infrastructure to run	Less governance control; third-party data dependency
Portkey	Managed + open-source option	Teams needing guardrails, conditional routing, request metrics	Hosted tier adds a platform fee
RouteLLM	Self-hosted (routing framework only)	Adding a learned routing brain on top of an existing gateway	Benchmark numbers must be re-validated on your own traffic

OpenRouter runs the routing layer for you, so there is no infrastructure to operate. LiteLLM runs inside your own infrastructure, so your data stays on your network and you pay no platform fee, in exchange for operating PostgreSQL, Redis, and Docker yourself.

The router itself adds latency — it has to look at the request before it can route it. That overhead is real but small relative to inference. Rule-based routing adds under 1 ms. Embedding-based routing adds about 5 ms. Semantic routing and heavier ML classifiers add 50–100 ms. For most applications that is acceptable. For a sub-100 ms voice interface it is not.

Token Waste: Where to Actually Look

A useful mental model separates prompt content into two groups. Quality-critical tokens carry task intent, constraints, data, and format requirements that materially change the answer. Waste-prone tokens repeat, decorate, or over-explain information the model already has.

Four categories worth auditing in rough order of impact:

1. Stale chat history. Most chat implementations append the full history on every turn. By turn ten, the first six turns are rarely relevant. Summarize or truncate. This is the most common source of runaway token counts in multi-turn agents.
2. Over-retrieved RAG context. Retrieve top-k, then rerank and trim to what the task actually requires. The retrieval tuning decisions that affect cost at scale are discussed in the pgvector vs Pinecone comparison.
3. Verbose system prompts. Remove repeated instructions, compress verbose system messages, use compact structured requirements, and keep only examples that change the answer.
4. Uncapped output length. Set max_tokens on every call. Output tokens are generated sequentially during the decode step, and decode is often memory-bandwidth-bound, so output length dominates perceived latency — and cost.

Batching: The Discount Most Teams Ignore

OpenAI's Batch API and Anthropic's Message Batches API both offer 50% cost discounts for async batch processing, with results delivered within 24 hours. On Anthropic's API, the Message Batches discount stacks with prompt caching discounts. If your batch requests share a large common context block, the cached input token discount applies on top of the 50% batch discount, compounding your savings.

If you run nightly document processing, bulk classification, or scheduled report generation, there is no reason to pay synchronous inference rates. The only trade-off is response time measured in minutes rather than seconds — which is fine for any workflow that is not user-facing in real time.

How This Applies to Production Work

The RAG and LLM systems I build through RAG and LLM development are designed with these cost levers in from the start, not retrofitted after the first invoice. Retrieval depth, model selection per task, and cache breakpoint placement are architecture decisions. The Biz365 AI build is an example of a production AI system where the inference cost envelope mattered directly to the unit economics of the product — retrieval scope and model tier for each pipeline step were scoped together, not independently.

FAQ

Does prompt caching work across different users' sessions?

On Anthropic, the cache is scoped per organization and per exact prefix. Two users hitting the same system prompt get a cache hit on that shared prefix, but their individual message histories are not shared. A large stable system prompt caches effectively across all users; a personalized prompt that changes per user does not.

What is a realistic combined reduction from these techniques?

A practical playbook combining smart routing, strategic caching, and batching of async work can reduce LLM spend 47–80% without degrading UX. Prompt caching alone can reduce API costs by 45–80% and improve time-to-first-token by 13–31%; semantic caching plus budget-aware routing achieves 47% spend reduction in production. The range is wide because it depends on traffic pattern: how repetitive the inputs are, what proportion are async-eligible, and how far apart your cheapest and most expensive model tiers sit.

When is a learned router worth the complexity over static rules?

Open frameworks like RouteLLM publish hard numbers: roughly 95% of frontier-model quality while sending only 14–26% of calls to the strong model, which lands as a 75–85% cost reduction on the routed traffic. Those numbers come from benchmark datasets. RouteLLM is the routing brain, not a gateway, with strong but benchmark-specific numbers you must re-prove on your own traffic. Static tiering with a confidence-based escalation gets most of the saving at a fraction of the implementation cost. Reach for a learned router when your task distribution is complex enough that rule-based classification makes too many escalation errors.

What should I track as an ongoing cost KPI?

Three numbers: cost per request by endpoint, cache hit rate per workload, and escalation rate if you are using tiered routing. Set per-request cost budgets and alert on overruns. A cost-per-request that creeps up without a corresponding increase in task complexity is usually a sign that context is growing unchecked somewhere in the pipeline.