Guardrails and evals for production agents
Putting an agent in production is harder than a single LLM call, because an agent is a loop with a multiplied failure surface — and you have to both contain it at runtime and measure it over time. A deep-dive on the two disciplines that make production agents trustworthy: guardrails that bound what an agent can do, and evals that test a whole trajectory, not a single output.
I’ve written before about what an agent is — function calling in a loop — and about evaluating non-deterministic LLM output. Putting an agent into production combines the hardest parts of both, and then adds new ones. An agent isn’t a single call you can wrap and check; it’s a loop where the model decides actions, runs tools, and steers across multiple steps, which means a much larger failure surface and a fundamentally harder thing to evaluate. Making a production agent trustworthy takes two distinct disciplines working together: guardrails that contain what the agent can do at runtime, and evals that measure whether it actually works over time. They answer different questions — “can this go wrong right now?” versus “is this getting better or worse?” — and you need both. This is a deep-dive on each, and on why an agent demands more of both than a single LLM call ever did.
Why an agent is harder than a single call
Start with the thing that makes everything else harder. A single LLM call has one input, one output, and one failure: it gives a bad answer. An agent has many steps, each building on the last, each potentially calling a tool that acts on the world — so its failure modes multiply and compound. A wrong decision at step two doesn’t just produce a wrong answer; it sends the agent down a path where every later step reasons from the mistake, possibly taking real actions along the way. That’s why an agent needs both tighter runtime containment (because the blast radius of a wrong action is larger) and a more sophisticated evaluation (because you’re judging a whole trajectory, not a single output). Hold that distinction — runtime containment versus over-time measurement — because the two disciplines map onto it.
Guardrails: containing the agent at runtime
Guardrails are the runtime controls that bound what an agent can do while it’s running, so that when it goes wrong — and it will — the damage is limited. They’re the seatbelt, active on every single run. The essential set, building on the agent-safety basics but hardened for production:
- Input guardrails. Validate and sanitize what goes into the agent. Filter prompt-injection attempts, reject malformed or out-of-scope requests, and constrain the task space before the loop even starts. The agent should never begin a run on input you haven’t vetted.
- Action gating. The most important guardrail. Tools that act — modify data, send messages, spend money, anything irreversible — must pass through a control point: human approval, or at minimum a hard policy check, before they execute. Read-only tools can run freely; consequential ones cannot run off the model’s unconfirmed decision. This is non-negotiable for an agent in production.
- Step and resource limits. A hard cap on iterations so a runaway loop fails after N steps instead of spinning forever; timeouts so a stuck agent doesn’t hang; and cost caps, because an agent is many metered calls and a runaway one is expensive while it’s useless. These turn “loops forever / costs unboundedly” into bounded, handleable failures.
- Output guardrails. Validate what the agent produces or is about to do before it reaches a user or a system — checking for inappropriate content, policy violations, or actions outside the agent’s remit. The last line before the agent’s output becomes real.
- Sandboxing and least privilege. Give the agent only the tools and access it genuinely needs, scoped as narrowly as possible, so the reach of any mistake is contained structurally. An agent that only needs to read can’t damage what it can’t touch.
The unifying principle is the one this series keeps returning to: the model proposes, your code disposes. Every consequential thing an agent “decides” passes through a control point you own, where you validate, gate, limit, or refuse. Guardrails don’t make the agent correct; they make its incorrectness survivable, which in production is what matters.
Evals: measuring the agent over time
Guardrails keep a run from doing damage, but they don’t tell you whether the agent is any good, or whether a change you made improved or degraded it. That’s what evals are for — and evaluating an agent is meaningfully harder than evaluating a single LLM output, because you’re not scoring one response, you’re scoring a whole trajectory: a sequence of decisions and tool calls toward a goal. The disciplines:
- Evaluate the whole trajectory, not just the final answer. An agent can reach a right answer through a terrible path — ten wasteful steps, a tool called wrongly then recovered from — or a wrong answer through reasonable-looking steps. You care about both the outcome and the path: did it accomplish the task, and did it do so efficiently and correctly along the way? Scoring only the final output misses half of what can be wrong.
- Measure task completion against a representative set. As with single-output evals, build a set of representative tasks — including the hard and adversarial ones — and measure how often the agent actually completes them correctly. This is your headline metric: does the agent accomplish what it’s for, across the range of inputs it’ll face?
- Keep a regression set of past failures. Every time the agent fails in a new way in production, add that scenario to the eval set, so a future change can’t silently reintroduce the failure. Over time this accumulated set of “ways the agent has broken” becomes the most valuable test asset you have — exactly as it is for single-output features, but more so, because agents have more ways to break.
- Track efficiency, not just correctness. Steps taken, tools called, tokens spent, time elapsed — because an agent that completes the task but takes twenty steps to do a three-step job is both expensive and fragile. Efficiency is a quality dimension for agents in a way it isn’t for a single call.
The shift from single-output evals is the move from scoring an answer to scoring a process. It’s harder, but it’s the only way to know whether your agent works and whether your changes help — and without it, you’re tuning a multi-step non-deterministic system blind, which is a recipe for an agent that quietly degrades.
How the two fit together
The crucial point is that guardrails and evals are complementary, not alternatives, because they answer different questions at different times. Guardrails operate at runtime, on every single run, bounding what this execution can do — they’re about safety and containment, ensuring no run causes damage. Evals operate over time, on a test set, measuring whether the agent works well and whether changes improve it — they’re about quality and progress. An agent with guardrails but no evals is safe but you have no idea if it’s any good or whether you’re making it worse; an agent with evals but no guardrails might score well on your set but cause real damage on the adversarial production input you didn’t test. Production agents need both: guardrails so failures are survivable, evals so you can actually improve the thing with confidence. Together they’re what turn an impressive agent demo into an agent you can responsibly run in front of real users and real systems.
Verdict
Putting an agent in production is harder than a single LLM call because an agent is a loop with a multiplied, compounding failure surface — and trustworthiness requires two distinct disciplines working together. Guardrails contain the agent at runtime, on every run: validate inputs, gate consequential actions behind approval or hard policy (the non-negotiable one), enforce step, time, and cost limits so runaway loops fail bounded, validate outputs, and apply least-privilege sandboxing so any mistake’s reach is contained. They don’t make the agent correct; they make its incorrectness survivable. Evals measure the agent over time against a representative set, and the key shift from single-output evaluation is scoring the whole trajectory — outcome and path — not just the final answer, tracking task completion, keeping a growing regression set of past failures, and measuring efficiency alongside correctness. The two are complementary: guardrails answer “can this run cause damage right now?” and evals answer “is this agent good, and are my changes helping?” — runtime containment versus over-time measurement. An agent needs both, because guardrails without evals is safe-but-unknown and evals without guardrails is good-on-paper-but-dangerous. Build both, and you have the discipline that turns an agent demo into a production agent you can actually trust — which, as always, is most of the real work.