Pre-launch evals tell you an agent works on your test set. Production tells you whether it works on reality — and only continuous monitoring closes that gap. A deep-dive on observing agents in the wild: what to log, what to measure, how to catch silent degradation, and how production failures feed back into your evals.
When Rails 8 shipped, I was enthusiastic about the no-Redis 'Solid' stack and the one-person-framework thesis. Now I've run it in production for six months. Here's the honest retrospective: where it delivered, where the rough edges were, and whether the simplification held up under real load.
I've championed pgvector as the sensible default for vector search. After running it in production, here's what the introductory tutorials leave out: index choice and its trade-offs, the recall-versus-speed dial, keeping embeddings in sync, and the operational realities that only show up at scale.
Putting an agent in production is harder than a single LLM call, because an agent is a loop with a multiplied failure surface — and you have to both contain it at runtime and measure it over time. A deep-dive on the two disciplines that make production agents trustworthy: guardrails that bound what an agent can do, and evals that test a whole trajectory, not a single output.
The prototype was magic. Then we shipped it to real users, and everything the demo had hidden came due at once — cost, latency, abuse, the API going down, and the unique problem of supporting a feature whose output you can't fully predict. A war story about the gap between an LLM demo and an LLM product.