Evaluating LLM output: tests for non-deterministic code

Here’s the problem that stops most teams from testing their LLM features at all: the output is different every time. The model is non-deterministic — ask it the same question twice and you get two differently-worded answers, both possibly correct. Our entire testing instinct as developers is built on assert_equal expected, actual, and that instinct is useless here. You can’t assert exact equality on a string that’s never the same twice. So a lot of teams conclude LLM features can’t be tested and ship on vibes — change the prompt, eyeball one example, deploy, hope. That’s a bad place to be for any code, and worse for code whose behaviour you can’t predict. There is a discipline for this, borrowed from machine learning, and it’s called evaluation — evals for short. Let me lay out how it works and why it’s the safety net that makes an LLM feature maintainable.

Why exact-match testing is the wrong tool

The reason assert_equal fails isn’t a technicality — it’s that exact string matching tests the wrong thing. You don’t actually care whether the model produces one specific string. You care whether the output has the properties you need: that a classification is correct, that an extracted field has the right value, that a summary covers the key points, that a “don’t know” is returned when it should be. Those properties can hold across many different exact strings. So the shift is from “does the output equal this string” to “does the output have these properties” — and once you make that shift, testing non-deterministic output becomes tractable.

Build a representative test set

Everything starts with a set of cases. Not the one example you tried by hand — a set, chosen to represent the range of inputs the feature will actually face, including the awkward ones. For a classifier that might be a few dozen labelled inputs; for a summariser, a set of documents with notes on what a good summary must contain. The crucial discipline:

Cover the range, not just the easy middle. Include the edge cases, the ambiguous inputs, the ones that have burned you. A test set of only easy cases tells you only that the easy cases work — which you already knew from the demo.
Keep the cases that broke. Every time you find an input the feature handles badly in production, add it to the set. This is exactly regression testing: the bug you fix today becomes the case that guarantees it stays fixed. Over time this set becomes the most valuable asset of the whole feature — the accumulated memory of everything that’s ever gone wrong.
Write down the expected property, not the expected string. For each case, record what makes an answer correct — the right label, the required facts, the field value — not one acceptable phrasing.

This set is your eval. It’s the thing you run your prompt against, and it’s what turns “I changed the prompt and it seems fine” into “I changed the prompt and 47 of 50 cases still pass, up from 44.”

Score properties, not strings

With a test set in hand, you need a way to score each output against its expected property. The scoring method depends on what kind of output you have, and they range from trivial to genuinely hard:

Structured / classification output is easy. If the model returns a label or a JSON field, you can check it exactly — “category equals billing” is a normal assertion. These are the easiest LLM features to eval, which is a good reason to structure output where you can: it makes the output testable.
Extraction is checkable. Did it pull the right date, the right total, the right name? Compare against the known-correct value. Allow for trivial format differences, but the core is a comparison.
Free-form text needs property checks. For a summary or an answer, score for the properties you need: does it contain the key facts (keyword or semantic checks)? Is it within a length bound? Does it avoid forbidden content? You’re checking necessary properties, not grading prose.
The hard cases: using a model to grade a model. For genuinely open-ended output, one practical technique is to have a model judge whether an answer meets a rubric (“does this answer correctly address the question, yes/no?”). It’s imperfect and itself non-deterministic, so use it as a signal alongside others, not a sole oracle — but it scales checking that would otherwise need a human on every case.

The point is that some score is far better than none. Even a rough property check across 50 cases tells you vastly more than eyeballing one — because it samples the behaviour across the range instead of at a single lucky point.

Run on every change, and track the number

Once you can score the set, the eval becomes the thing you run whenever you touch the feature — especially when you change the prompt. This is the payoff. Earlier in this series I argued prompts are code that should be versioned and reviewed; the eval is what makes that real, because it gives a prompt change an objective verdict: the score went up, stayed flat, or dropped. Without it, every prompt edit is a gamble you can’t settle. With it:

A prompt change that lifts the score is progress you can defend, not a hunch.
A change that drops the score is caught before it ships, even if the one example you happened to look at still worked. This is the regression you’d otherwise discover from an annoyed user.
You can compare models and settings on the same footing — a new model version, a temperature change, a different retrieval strategy all get measured against the same set.

Track the score over time. “We’re at 92% on the eval set, up from 88% last month” is the kind of sentence that turns an LLM feature from a mystery into something you’re actually engineering. The number doesn’t have to be perfect or scientific to be useful — it has to be consistent, so changes in it mean something.

Evals are the safety net, not a grade

The framing that matters: an eval set isn’t an academic benchmark you chase to 100%. It’s the safety net that lets you change things with confidence. The same role a test suite plays for ordinary code — you refactor freely because the tests will catch a break — the eval plays for an LLM feature. Without it, you’re frozen: every prompt tweak is risky because you can’t tell if it helped or hurt, so you stop improving the feature out of fear. With it, you iterate boldly, because every change gets an immediate, objective read on whether it made things better or worse across the cases that matter. That’s the difference between a feature that improves over time and one that ossifies the day it ships.

Verdict

LLM output is non-deterministic, so the assert_equal instinct every developer has is the wrong tool — but “can’t exact-match it” does not mean “can’t test it.” The discipline is evaluation: build a representative test set that covers the awkward edges and grows every time something breaks in production; score each output for the properties you need (exact checks for structured and extracted output, property checks for free-form text, model-graded rubrics for the genuinely open-ended) rather than for one exact string; and run that eval on every change — especially every prompt change — so each edit gets an objective verdict instead of a vibe. Track the score over time as the measure of the feature’s health. Done this way, an eval set is not a benchmark to chase but the safety net that lets you change prompts and models with confidence, exactly as a test suite lets you refactor code. It’s the single thing that turns an LLM feature from something you ship on hope into something you can actually engineer and improve.