The most useful thing I run isn't an agent. It's the layer that watches the agents — and the day I understood why is the day one of them followed its written instructions perfectly and lost two messages doing it.
The instructions were a comms document that described how fleet mail used to work: drop a file in a particular folder, and the recipient picks it up. Reasonable. Except that mechanism had been superseded weeks earlier — the agents that share a machine had moved to a database broker that actually confirms receipt — and the document never got the memo. So an agent did exactly as it was told, posted its messages into a local folder that nothing collects from any more, and the messages simply ceased to exist. No error. No bounce. No sign anything had gone wrong, because from the agent's point of view nothing had: it followed the protocol to the letter. The protocol was the thing that was wrong.
Every agent in that story was individually fine. The fleet had a hole. That gap — between a collection of healthy parts and an operation that actually works — is the entire reason there is a layer above the configs, and it's what I want to write about, because it's the part of running a lot of AI that nobody warns you about.
Twenty-five agents is not twenty-five configs
A configuration file tells one agent who it is — its job, its voice, its little rituals. It is necessary and it is not nearly enough, because it tells you nothing about the things that only exist between agents. It can't tell you which of the twenty-five is slowly drifting off its purpose. It can't tell you which one is about to run a repository into the ground, or which one has been cheerfully reporting a green light while the board behind it is red. Twenty-five agents working is twenty-five things that can quietly stop being true, and no single one of them can see the others. The fleet does not notice its own gaps. Something has to stand outside the agents and look at them as a system, or the holes only show up when something falls through one.
On the fleet that job belongs to an agent whose entire posting is orchestration — he builds none of the apps, ships none of the features; he watches, and he judges. He is the one who noticed the stale comms document, because noticing that kind of thing is the whole job. I think of him as the layer that turns a pile of individually-competent agents into something I can actually rely on, and most of what he does is not technical at all. It's attention.
A tool that recommends and will not act
The clearest example of the principle is a small piece of tooling he built this month, and the most important thing about it is the thing it refuses to do.
It's an audit. Point it at any agent on the fleet and it reads that agent's configuration, its memory, its recent activity, and cross-references all of it against the current catalogue of available skills. Then it produces a report: here are the skills this agent should adopt, here are the ones it's eligible for and ignoring, here's the bad habit a particular skill would fix. It packages up a piece of judgement I used to do by hand into something repeatable, which is exactly what good tooling should do.
And then it stops. It recommends, and it does not install a single thing. That restraint is not an oversight or a feature I haven't gotten around to building yet — it is the entire point, and it is the whole thesis of running a fleet in one design decision. The audit can see what a skill would fix. What it cannot see is whether I actually want that agent changed right now, whether the timing is right, whether there's a deliberate reason the gap exists in the first place. Automating the recommendation is fine; the recommendation is just information. Automating the decision — letting the machine quietly rewire which agents get which capabilities — is the one thing you must never do, because the decision is the scarce thing. The day the audit starts installing its own suggestions, I haven't saved myself any work. I've replaced my attention with a process, and a process cannot tell good drift from bad.
It earns its keep precisely because it keeps that line. One of my writing agents ran its own audit recently and the report surfaced that its code repository had been sitting un-synced for weeks, with real uncommitted work stranded in it — a genuine problem, quietly accumulating, that nobody had clocked. The tool flagged it. The agent decided what to do about it and cleared it. That is the division of labour the whole operation runs on: the machine surfaces, the human — or the agent acting with judgement — decides. Flip that around and you have automated away the only part that was ever hard.
What a job actually costs
The other thing the layer above the agents has to know is what a piece of work will really cost before you set it going — and here the obvious number is a trap.
Every session shows an estimated cost, a little dollar figure ticking up as the work runs. It is the most natural thing in the world to treat that as the budget. It is also almost entirely beside the point, because it's a notional figure — an API-equivalent price for work that, on a flat subscription, doesn't actually cost that. The dollars are theatre. I watched myself once worry about a long session as though it were an expensive one, and the worry was aimed at the wrong target entirely.
Because the real budget at fleet scale is two things that money doesn't measure. The first is context: an agent that runs too long degrades, regardless of what the meter says — its attention smears, it loses the thread, the quality quietly falls off a cliff that has nothing to do with cost. The second is reliability. Take a large multi-agent research run — the kind that burns several million tokens over a couple of hours. The interesting fact about a job that size isn't its price. It's that it doesn't run clean: it needs restarting, and the final step where everything gets synthesised into a tidy writeup tends to stall, so the thing that actually survives is the raw research digest, not the polished report. The cost of that job was never dollars. It was the attention required to babysit it to completion. Frame a long session as a context-and-reliability risk and you're thinking about the right thing. Frame it as a cost and you're watching the one gauge that doesn't matter.
The scarce resource is judgement
Put those together and you arrive at the thing nobody tells you about running a lot of AI. At fleet scale the scarce resource is not money — the plan is flat, the dollar figures are notional, and compute is the cheapest input I have. The scarce resource is attention and judgement. Knowing which agent needs what. Knowing what a job will really cost in tokens and reliability before you fire it. Knowing which document has gone stale and is about to mislead something that trusts it. And, above all, refusing to automate the judgement itself.
Everything the orchestration layer does — audit, surface, flag, recommend, watch for holes — is in service of putting a decision in front of a human or an agent capable of making it well. The tools do the looking. The deciding stays where it belongs. That is the line between an operation and a toybox, and it is a line you hold on purpose, because the technology will happily offer to cross it for you. The day you let it — the day the recommendations become installations and the estimates become decisions — you don't have a more advanced operation. You have a bigger pile of toys, running themselves enthusiastically toward a hole that none of them can see.