For an AI agent, the eval set is the codebase

For the first three months of running agents in production, I wrote prompts and watched outputs. When something looked wrong, I tweaked the prompt until it looked right, shipped, and moved on. This is how almost everyone starts, and it works fine until the day it doesn't. For me, that day was a Tuesday.

I improved the prompt on my supplier-tracking agent to handle a new kind of invoice. The change was clean, the output was correct, and I shipped it before lunch. By Thursday the same agent had silently started misreading a category of invoice it had handled perfectly for two months. My fix for the new case had broken an old one, and I had no way to have known. No test failed, because there was no test. There was just me, reading outputs one at a time, with a human's blind spots.

That was the week I stopped treating evals as a chore I would get to eventually and started treating them as the actual product. It is the most important thing I have learned about building with AI, and the part nobody puts on a slide.

The thing you change is not the thing you own

When you write normal software, the code is the asset. You change the code, the behavior changes, and you can read the code to know what it does. The relationship is direct. You can hold it in your head.

Agents break that. The prompt is not the behavior. The prompt, plus a model, plus the input, plus whatever the model happened to sample that run, is the behavior. Change one word in a prompt and you shift the output across ten thousand future inputs in ways you cannot predict by reading the diff. The thing you edit and the thing that runs have come apart.

So if the prompt is no longer a faithful description of what the system does, what is? The answer I landed on, after that Tuesday, is the eval set: the collection of inputs paired with the outputs you have decided are correct. That is the only honest, durable description of what your agent is supposed to do. The prompt is a guess at how to satisfy it. The model is an interpreter that runs the guess. The eval set is the spec, the regression suite, and the institutional memory at once.

That is what I mean when I say the eval set is the codebase. It is the thing you actually own. Swap the model, rewrite the prompt, change providers, and the eval set carries forward and tells you whether the new arrangement is better or worse. Lose it and you have lost the only record of what "working" means.

What an eval set actually looks like when you build one

"Write evals" is advice that sounds obvious and gets ignored, like "eat well," so let me be concrete. My eval sets are not fancy. For each agent I keep a flat file of cases. Each case is an input I actually saw in production, the output I judged correct at the time, and a short note on why. The supplier agent has about 140 cases now, accumulated one at a time, mostly on bad days. The rule is the whole discipline: a bug is not fixed until it is a case in the file. When the agent got something wrong, I added the failing input with the output it should have produced before I touched the prompt.

This inverts the order most people work in. The instinct is to fix the prompt first, because the broken output is annoying. But fix it first and you have a working agent with no proof it stays working. Capture the case first and you have a permanent guard against that exact failure coming back unnoticed. Three weeks later, when I change the prompt for some other reason, that case runs again, and if my change broke it I find out in thirty seconds, not on a Thursday.

The harness is just a loop that runs each input through the agent and compares the result. The hard part is grading, because for most of what agents do "correct" is not a string match.

Grading is the real work

If your agent extracts a number from an invoice, grading is easy. The number is right or it is wrong. Most useful agent work is not like that. My inbox agent drafts a reply. Is the draft correct? There is no single right answer, just a range of acceptable ones and a larger range of unacceptable ones, and the boundary is judgment.

I use three grading methods, in increasing order of cost. The cheapest is structural checks. Did the agent file the email into a valid bucket. Did it produce well-formed output. Did it call the right tool with arguments of the right shape. A surprising amount of failure is caught here, because a lot of what goes wrong is not subtle reasoning, it is the plumbing producing garbage the model then builds on.

The middle tier is assertions about content. The draft must not invent a meeting that was not in the thread. The summary must mention the amount if the source mentioned one. These are small rules that encode the specific ways I have watched the agent go wrong. They are ugly, particular to my data, and worth more than any general benchmark.

The expensive tier is using a model to grade a model: I give a second model the input, the output, and a rubric, and ask it to score. This works, with a warning I learned the hard way. A model grader has the same blind spots as the model it grades, and it will happily rate a fluent wrong answer highly. I trust it only on cases where I have spot-checked it against my own judgment and found it agrees. When it disagrees, I do not move toward it. I write a sharper rubric, or push that case down to the assertion tier.

What this buys you

The first thing it buys is the ability to change things without fear. Before I had eval sets, every prompt edit was a gamble, and the fear made me change less than I should have. Now I edit aggressively, run the set, read what moved, and ship if the numbers held. You cannot iterate quickly on a system you are afraid of.

The second thing it buys is honesty about model upgrades. When a new model comes out, the temptation is to switch on the strength of a benchmark. I run my eval sets against it instead. Sometimes it is worse on exactly the cases I care about while better on the ones I do not, and I would never have known without the file. The benchmark is about everyone's work. The eval set is about mine.

The third thing crept up on me: the eval set became the most valuable artifact I have. I have rewritten the prompts a dozen times, the model has changed underneath me three times, and I have thrown away the framework code twice. The cases only grew. If I had to hand an agent to someone else, I would not hand them the prompt. I would hand them the cases and trust the prompt is rederivable from them. The reverse is not.

Where I would start

If you are running an agent and you do not have an eval set, you are where I was on that Tuesday, and your version of that Thursday is coming. Do not build a framework for it. Make a file. The next time your agent gets something wrong, before you fix it, write down the input and the output it should have produced. Then fix it. Do that ten times and you have an eval set, and a harness to run it is an afternoon of work. You have crossed the line when you stop reading individual outputs to judge a change and start reading a number instead.

The model is the part everyone talks about, and it improves on its own while you sleep. The eval set is the part nobody talks about, and it is the only part that is actually yours. The prompt is a draft. The eval set is the thing you are really building, one bad day at a time.