Cost-control patterns for LLM-heavy apps

Once an app leans heavily on LLM calls, cost stops being a bill you receive and becomes a dimension you engineer. Because every call is metered per token, spend scales directly with usage, and an app that does a lot of LLM work can run up a bill that dwarfs its other infrastructure combined. I learned some of this the expensive way, in production. The good news is that cost in an LLM-heavy app is controllable with a set of well-understood patterns — the same way latency or memory is controllable: you measure it, you understand where it goes, and you apply the right techniques. This is a deep-dive on those patterns, in the order you should actually apply them, starting with the one that has to come first.

Measure before you optimize

The first pattern isn’t a cost-reduction technique at all — it’s measurement, and it’s non-negotiable because everything else depends on it. You cannot control what you cannot see, and the single most common mistake is trying to cut cost before knowing where it goes. So before any optimization:

• Track token spend per feature and per user. Know which features consume the most tokens and which users drive the most usage. Cost is almost never uniform — a few features and a few heavy users typically dominate, and you can’t find them without instrumentation.
• Break down input vs output tokens. They’re often priced differently, and a feature might be expensive because of huge inputs (stuffing too much context) or huge outputs (asking for more than you need). The fix differs depending on which, so you have to see the split.
• Watch the distribution, not just the average. The average call hides a long tail of expensive ones. It’s usually the tail — the giant document, the runaway agent loop — that drives the bill, and averages conceal it.

With this visibility, optimization becomes targeted: you attack the features and patterns that actually cost money, instead of guessing. Without it, you’re optimizing blind. Measurement first, always.

Pattern 1: model tiering

The highest-leverage cost lever is usually using the right-sized model for each task. The most capable models cost dramatically more per token than smaller, faster ones — and a great many tasks don’t need the most capable model at all. So tier your model usage:

• Use a cheap, fast model for easy tasks — classification, simple extraction, routing, formatting. These are well within a small model’s ability, and paying flagship prices for them is pure waste.
• Reserve the expensive model for tasks that genuinely need it — complex reasoning, nuanced generation, the hard cases where the quality difference actually shows up in your evals.
• Consider a routing step — sometimes a cheap model (or simple logic) can decide whether a request needs the expensive model at all, escalating only when necessary. The cheap triage pays for itself many times over.

This single pattern often cuts cost more than everything else combined, because the price gap between model tiers is large and the proportion of tasks that need the top tier is usually small. The discipline is the same match-the-tool-to-the-need one this series keeps returning to — don’t reach for the biggest model reflexively; use the smallest one that passes your evals for that task.

Pattern 2: caching

Many LLM-heavy apps make the same or similar calls repeatedly, and paying full price for a response you already generated is waste. Caching attacks that:

• Exact-match caching. If the identical input comes in again, return the stored response instead of calling the model. For deterministic, repeatable requests (a fixed prompt over fixed data) this is straightforward and can eliminate a large fraction of calls. The cheapest call is the one you don’t make.
• Semantic caching. More advanced: if a semantically similar request has been answered before (found via embedding similarity, the pgvector technique repurposed), you can return or adapt the cached answer. This is powerful for things like FAQs where many users ask the same question differently — but it needs care, because “similar enough to reuse” is a judgment call and a wrong cache hit serves a subtly off answer.
• Provider-side prompt caching. Some providers let you cache a large, reused prompt prefix (a big system prompt, a shared context) so you’re not billed full price for re-sending it every call. If you’re sending the same large context repeatedly, this is close to free money.

Caching is the classic “don’t repeat expensive work” optimization applied to LLM calls, and in apps with repetitive usage it can be the second-biggest lever after tiering.

Pattern 3: token discipline

Since you pay per token, the size of every prompt is a cost, and a lot of LLM-heavy apps are casually wasteful with tokens. Tightening this is steady, broad-based savings:

• Trim the context you send. Don’t stuff the whole document when the relevant section will do; don’t include the entire conversation history when a summary suffices. In RAG especially, retrieving and sending fewer, better chunks cuts input tokens directly — and often improves quality by reducing noise, so it’s a double win.
• Constrain output length. Ask for only what you need. If you need a category, don’t let the model write a paragraph explaining it. Shorter, structured outputs cost less and are usually more useful anyway.
• Tighten prompts. Verbose, rambling prompts cost tokens on every single call. A concise prompt that works as well is strictly cheaper, multiplied across all your traffic.

Token discipline is less dramatic than tiering or caching per call, but it applies to every call, so the aggregate saving across an LLM-heavy app is substantial — and it frequently improves quality and latency as a side effect.

Pattern 4: budgets and limits

Finally, the defensive pattern — the one that protects you from the cost surprises rather than reducing steady-state spend. Because usage isn’t uniform and some of it is adversarial or accidental, you need guardrails:

• Per-user rate limits and budgets. Cap how much any single user can consume in a window, so one heavy (or abusive) user can’t run up an unbounded bill. This is the limit I most wished I’d had before production taught me.
• Hard caps on expensive operations. Limit the size of inputs you’ll process, the number of steps an agent can take, the length of outputs. Each is a ceiling on the worst-case cost of a single operation.
• Alerting on spend. Know quickly when cost spikes — an alert when daily spend exceeds a threshold turns a runaway bill from an end-of-month shock into a same-day fix.

Budgets and limits don’t reduce your normal cost; they bound your abnormal cost, which is exactly the part that produces the nasty surprises. They’re cheap insurance against the tail.

Verdict

In an LLM-heavy app, cost is an engineering dimension you design for, not a bill you discover — because metered per-token pricing means spend scales directly with usage. The patterns that control it apply in a clear order. Measure first: track token spend per feature and per user, split input from output, and watch the expensive tail, because you can’t optimize what you can’t see. Then model tiering — the highest-leverage lever — using a cheap model for easy tasks and reserving the expensive one for tasks that genuinely need it, often the biggest saving of all. Caching eliminates repeated work (exact-match, semantic, and provider-side prompt caching), the cheapest call being the one you never make. Token discipline — trimming context, constraining output, tightening prompts — is modest per call but applies to every call and frequently improves quality too. And budgets and limits — per-user caps, ceilings on expensive operations, spend alerts — bound the abnormal cost that produces the nasty surprises. The through-line is the one this series keeps making: treat the LLM API as the metered external service it is, and engineer its cost as deliberately as you engineer latency or memory. Do that, and an LLM-heavy app’s bill goes from an alarming mystery to a controlled, understood line item — which is the difference between a feature you can afford to scale and one that scales you into trouble.