Why I kill agents that work

In May I shut down an agent that had not made a visible mistake in eleven weeks. Every evening it read my calendar, pulled the history on everyone I was meeting the next day, and left a tight brief waiting for me at seven in the morning: who they were, what we had discussed before, what they probably wanted from the call. By any reasonable measure it was working. That is exactly why it took me three months to notice it had to go.

Everyone who runs agents in production talks about the ones that fail. Failure is the easy case. It announces itself, it shows up in the logs, it pages you at a bad hour and demands a fix. Nobody talks about the discipline of killing the ones that work, and after enough time running a fleet, I think that discipline is most of the difference between a stack that compounds and a stack that quietly rots.

The fleet grows faster than the attention

Here is the trap, and it is built out of good news. Once the glue is in place, the identity plumbing, the observability, the escalation channels, a new agent costs an afternoon. My first agent took three weeks to build. My fourth took two hours. I have written about that curve as the reason this wave of automation is different, and I stand by it. But the curve has a second half that nobody mentions: when building gets that cheap, you build too much. By late spring I was running eleven agents. Every one of them was doing something. Barely half were doing something that mattered.

An agent is not a script that runs and exits. It is a standing process with credentials, a token budget, write access to some corner of your life, and a permanent claim on your future attention. It has to be watched, evaluated, re-checked when the model underneath it changes, patched when a tool it depends on shifts shape. The cost of building an agent has collapsed. The cost of owning one has not moved. Attention is the real budget line, and by May mine was overdrawn.

Working is not the same as watched

The reason working agents worry me more than broken ones comes down to one asymmetry. A broken agent gets attention. A working agent earns trust, and trust is the process by which you stop checking.

I have written before that the eval file is the real asset when you run agents: the growing set of cases that defines what correct means for each one. The corollary I have learned since is that the eval file is also a vital sign. When an agent's file is growing, someone is watching it, catching edge cases, disagreeing with outputs, feeding judgment back into the system. When the file freezes, the watching has stopped. The drift has not. Prompts rot, models update underneath you, the data changes shape, and the gap between what the agent does and what you believe it does widens in silence, at precisely the moment you are most confident in it.

The most dangerous object in my stack is not the flaky experiment I check five times a day. It is the reliable agent with write access that I trust completely and have not really read in a month.

Three kills, three reasons

The meeting brief agent died first, and the reason took me the longest to admit. The briefs were accurate. They were also making me worse. I was walking into conversations holding a summary where my questions used to be. It turns out that preparing for a meeting is not overhead wrapped around the thinking. It is the thinking. Digging through the history yourself is how you arrive with an opinion instead of a recap. The agent was doing my learning for me, and learning is not a task you can delegate and still have. Some work is the job.

The second kill was a sub-agent on the construction project my supplier agent tracks. Its whole purpose was chasing delivery dates: polite reminder emails, follow-ups, escalation to me after two silences. It worked. Deliveries got confirmed, gaps got flagged. It was also automating around a broken arrangement. The honest fix was a single consolidated weekly status from the site manager, which took one slightly awkward phone call to arrange. When you automate a broken process you do not fix it, you preserve it. The agent was making the brokenness sustainable, which is worse than leaving it painful, because pain at least gets things changed.

The third was a digest agent that watched funding announcements and model releases across the areas I invest in and produced a clean morning summary. Killing it required one question: what did I do differently in the last month because of this output? The answer was nothing. I skimmed it, felt informed, and moved on. Not one meeting, one pass, one check, one email traced back to it. An agent whose output changes no decision is not working, whatever its accuracy says. It is producing exhaust with good formatting.

The quarterly kill review

Those three deaths turned into a ritual. Once a quarter I sit down with the fleet and ask four questions about each agent, in this order.

What did I do differently in the last thirty days because of it? Not what it produced. What changed. If I cannot name something specific, the agent is a cost with a status page.

Is its eval file still growing? A frozen file means nobody is watching, and an unwatched agent is drifting by default. Either I recommit to watching it or I admit it does not matter enough to watch, and an agent that does not matter enough to watch does not matter enough to run.

Is it holding a decision I should be keeping? Some loops exist to produce an outcome, and those I automate without guilt. Some loops exist to keep me sharp: reading the hard email myself, preparing for the conversation myself. Automating those is trading capability for convenience at a terrible exchange rate.

Would I build it today? This is the sharpest one, because the answer used to be distorted by sunk cost. It is not anymore. A rebuild costs an afternoon, and the rebuilt version is usually cleaner, on a current model, with the eval cases carried forward. Anything worth having back can come back better in a day. Killing is nearly free now. Most people's defaults were formed when it was expensive, and they have not caught up.

The same question, pointed at companies

I have started asking founders a version of this in diligence. Teams love to tell me how many internal agents and automations they run. The number is a vanity metric. The question that separates operators from collectors is: what did you decommission last quarter, and why? A team that only ever adds automation is not managing a fleet. It is accumulating one, and the interest on that accumulation compounds just as reliably as the leverage does. The teams that can name their kills, and the reasoning behind them, are the ones whose stacks I would bet on three years out.

My fleet is down to six agents. The stack got smaller in number and larger in what it actually carries, and every remaining agent has a current eval file and a recent answer to the thirty-day question. That is the state I now try to hold, one kill at a time.

An agent that works is not a reason to keep it. It is only the reason the decision is hard.