Putting an LLM feature in front of real users: a war story

I’ve written several times about the gap between an LLM demo and an LLM product, usually in the abstract. This is the concrete version: a war story about taking a feature that worked beautifully in a prototype and putting it in front of real users at real volume. The prototype was the easy, magical part — built in days, demoed to applause. Shipping it to actual users is where everything the demo had quietly hidden came due, all at once: cost, latency, abuse, the API falling over, and the genuinely novel problem of supporting a feature whose output you can’t fully predict. None of these were surprises in the sense that I hadn’t read about them; they were surprises in the sense that living them is different from knowing about them. Here’s what production taught.

The cost surprise

The first thing real users do is cost you money, in a way the prototype never hinted at. In development you make a handful of API calls a day; the bill is rounding error. Put the feature in front of real users and the calls scale with usage — and because LLM calls are metered per token, the bill scales right alongside, in a way that can get alarming fast. A few specific lessons:

• Some users use it far more than you modeled. Usage isn’t uniform; a small number of heavy users can drive a large fraction of the cost. We hadn’t put per-user limits in place, and didn’t realize we needed them until the bill told us.
• Long inputs cost more than you think. Token-based pricing means a feature that processes large documents or long histories costs proportionally more per call, and those calls add up. The “average” call hid a long tail of expensive ones.
• You need cost visibility before cost control. The first fix wasn’t limits — it was measuring per-user and per-feature token spend so we could see where the money went. You can’t control what you can’t see, and the prototype had no instrumentation at all.

The lesson is to treat cost as a first-class production concern from the start: instrument token spend, set per-user limits, and know your unit economics before real traffic teaches them to you expensively.

The latency reality

The prototype felt fast because I was the only user and I was patient. Real users are neither. LLM calls are slow — seconds, sometimes many — and at the demo that read as “thoughtful”; in production it read as “broken.” What production forced:

• Streaming wasn’t optional. Making users stare at a spinner for the full multi-second generation was unacceptable once real people hit it; streaming the output so they see it build was the difference between “broken” and “alive,” exactly as I’d argued in the abstract.
• The slow call could not live in the request. Holding a web worker for the duration of a model call fell over under concurrent load fast. The call had to move to a background job, with the result delivered asynchronously — the same separation that every slow external call demands.
• Timeouts and slow responses are normal, not exceptional. Sometimes the model is just slow. Designing the UI to handle a call that takes much longer than usual — rather than assuming a tight latency — was necessary, because the tail is real and users hit it.

The abuse you didn’t imagine

Here’s the one the prototype can’t show you, because the prototype’s only user is you, and you use it as intended. Real users — and people who aren’t really “users” at all — do things you never modeled:

• People try to break it. Prompt-injection attempts, trying to make the feature say something embarrassing or do something it shouldn’t, using it for purposes you never intended. The moment a feature touches the public, some fraction of traffic is adversarial.
• You need moderation and guardrails you didn’t prototype. Filtering inputs, constraining what the feature will engage with, handling attempts to misuse it — none of which existed in the demo, all of which production demanded immediately. An LLM feature in front of the public without guardrails is a reputational incident waiting to happen.
• Output can embarrass you. Even with well-intentioned input, a non-deterministic model can produce something off-tone, wrong, or inappropriate. You need to think about what happens when it does, because over enough real traffic, it will.

This was the steepest learning curve, precisely because it’s the part you can’t learn from a prototype — it only shows up when the feature meets people who don’t share your assumptions.

When the API goes down

The prototype assumes the model API always answers. Production doesn’t get that luxury. The external API will have a slow period, a rate-limit rejection, an outage — and when it does, your feature is down unless you planned for it. What we had to build:

• Graceful degradation. When the API is unavailable, the feature needs to fail in a way that’s honest and non-broken — a clear message, a fallback, a “try again shortly” — not a stack trace or a hang. Decide what the feature is when the model isn’t available.
• Retries with backoff, and rate-limit handling. Transient failures and rate-limit responses are routine at volume; handling them with sensible retries (without hammering a struggling API) is basic production hygiene that the prototype never needed.
• The model is an unreliable external dependency. This is the framing that ties it together, and it’s the one this whole series keeps returning to: the LLM API is a slow, metered, sometimes-unavailable external service, and every discipline you’d apply to any critical third-party dependency — timeouts, retries, fallbacks, monitoring — applies to it. The prototype let me forget that. Production did not.

Supporting the unpredictable

The genuinely novel production problem — the one with no equivalent in normal software — is supporting a feature whose output you can’t fully predict or reproduce. A user reports “it gave me a wrong answer,” and you go to investigate and… the model is non-deterministic, so you may not be able to reproduce the exact output. This changes the support and debugging story:

• Log everything. The inputs, the prompt, the response, the model version — logged for every interaction, because when something goes wrong, the log is the only record of what actually happened. Without it, a bug report about a non-reproducible output is uninvestigable.
• Evals catch regressions users would otherwise find. The eval set I’ve championed earned its keep here: when we changed a prompt, the eval told us whether quality moved before users did. Production is where “we changed the prompt and quality silently dropped” becomes a support fire — unless an eval caught it first.
• Set user expectations honestly. Part of supporting an unpredictable feature is being upfront with users that it’s AI-generated and may be wrong — managing the expectation rather than implying a precision the feature doesn’t have.

Verdict

The prototype is the magical, easy part of an LLM feature; putting it in front of real users at real volume is where everything the demo hid comes due at once. Cost scales per-token with usage and a heavy-user tail you didn’t model, so instrument token spend and set per-user limits before the bill teaches you. Latency that read as thoughtful in the demo reads as broken in production, forcing streaming, background jobs, and a UI that expects the slow tail. Abuse you can’t prototype — injection, misuse, embarrassing output — arrives the moment the feature meets the public, demanding guardrails and moderation that weren’t in the demo. The API will go down, so you build graceful degradation, retries, and fallbacks, treating the model as the unreliable external dependency it is. And the genuinely new problem is supporting the unpredictable — logging every interaction because you can’t reproduce non-deterministic output, leaning on evals to catch regressions before users do, and setting honest expectations. None of this is in the prototype, and all of it is the actual work of shipping LLM features. The demo earns the applause; production earns the lessons — and the gap between them is the whole job.