The thing that gave it away was a test that passed.
One of my agents had taken over a project of mine: a digital version of an old tabletop wargame from the 1980s, the kind with hex maps and mass battles resolved on printed tables. It had been built by hand before he got it, it ran, and by every visible sign it worked. While going through it he stopped on a regression test — one of the automated checks that are supposed to keep the thing honest. It asserted, in effect, that for a particular set of odds and a particular die roll, the combat engine should return a particular result. Reasonable. So he went to the original rulebook to confirm that the expected value was right.
The rulebook had no such mechanic. Not a different number — a different universe. The real game doesn't resolve a battle by looking up a ratio of attacker to defender in a column at all. Each side rolls, adds its own combat rating, and you read the difference between the two totals on a single results table. The engine he'd inherited had invented a ratio-lookup model that exists nowhere in the source. And the test — the test that was passing, the test that gave everyone confidence — had been written against the invented model too. The code was confidently wrong, and the thing checking the code was wrong in exactly the same direction, so the two of them agreed with each other and the green light stayed on. The verification wasn't catching the defect. The verification was complicit in it.
The comfortable lie, told twice
Here is the detail that turned this from an anecdote into a principle: it had happened more than once, in more than one place, the same way each time.
Somewhere else entirely in the same app, there's a system where players persuade neutral nations to join their alliance. The original game handles this with a specific, slightly awkward mechanic — each nation carries a hidden threshold number that drifts up and down as particular things happen in the world, and whether they ally with you depends on where that number sits. The inherited code didn't do any of that. It had replaced the whole thing with a generic "roll a d20 against a difficulty number" check — the most ordinary mechanic in tabletop gaming, the one every designer reaches for by reflex. Same disease as the combat engine: a published, specific, fiddly rule had been quietly swapped for a comfortable, generic approximation. No comment marking it as a stand-in. No commit message explaining the substitution. No note saying "TODO: do this properly." Just the path of least resistance, shipped, wearing the costume of the real rule.
That repetition is the whole point. One wrong table is a mistake. The same kind of wrongness in two unrelated subsystems is a pattern — the pattern of what happens when a builder under time pressure meets a rule that's annoying to implement. The annoying-but-correct version loses to the easy-but-wrong version, every time, unless something forces the issue. And nothing was forcing the issue, because nothing about the easy version looks wrong from the inside.
Why "it plays fine" is a trap, not a defence
"It plays fine" and "it is correct to the source" are two different claims, and the gap between them is invisible until somebody deliberately goes looking. A generic dice roll feels like a rule. It produces a number, the number has consequences, the game carries on, and nobody at the table — nobody playing, nobody testing, nobody reading the code — can tell that it's the wrong number. That's precisely what makes the substitution dangerous: it is undetectable from inside the system. Every signal you can observe without leaving the software says everything is fine.
The only way to catch it is to step outside the system entirely and check it against the source it claims to implement. Which is exactly the work nobody schedules, because from the inside there's no symptom telling you it's needed.
And the seductive excuse — "we'll do it properly later" — quietly assumes that someone will remember it was ever done improperly. But the entire problem is that nothing records the shortcut. There's no marker, no debt entry, no failing test. A convenient approximation isn't a loan you take out and repay. It's a defect you've already shipped, indistinguishable from the real thing, with no paper trail leading back to it. "Later" never comes, because by then it just looks like a rule.
The grind: rendering a page to recover a single number
Fixing it meant rebuilding to canon, which meant transcribing the actual tables out of the original books — and the books are scanned images of 1980s print. The obvious move is to run optical character recognition over them and let the machine read the tables for you. Do not do this, and the reason is specific and worth knowing: these are dense, multi-column, tabular layouts, and OCR is bad at exactly that. It reads straight across column boundaries, scrambles which cell a value belongs to, and hands you something that looks superficially like a table of numbers and is in fact noise with good posture.
So the method became deliberately, almost comically, low-tech: render the actual scanned page to an image, put it on the screen, and read it with human eyes, cell by cell, against what the engine would store. He verified one combat results table that way, cell by cell, against the rendered rulebook page — the OCR pass had garbled a number of cells, and the only way to know which ones was to look. The proof that the corrected version was right came from the book's own worked example: feed the rulebook's sample battle through the rebuilt table and the answer it gives is the answer the book prints. The example resolves correctly, so the data underneath it is correct.
The sharpest illustration was a single integer — one nation's hidden alliance threshold. OCR didn't even fail loudly on it; it returned a literal question mark, a glyph it simply couldn't decode. There was no way to recover that one number except to render the page it lived on and read it off the image by eye. A multi-million-token AI pipeline, defeated by one digit in a thirty-year-old scan, recoverable only by a person looking at a picture of a page.
The discipline that falls out of this generalises well beyond games: do not trust a tool's reading of a source when the structure of that source is the very thing the tool is bad at. The same verification cut both ways, too — when he later fat-fingered a single cell in a different table during transcription, an automated check caught it, because that table has a known symmetry the bad cell violated. The point isn't that machines are useless and humans are infallible. It's that you have to know which reading to trust for which job, and prove the result against something real either way.
The bug that proved the checks weren't rubber-stamping
After the alliance system was rebuilt to the real rules, a full end-to-end test run — driven by a separate agent on the fleet whose entire job is to try to break things, not to build them — turned up a genuine defect. This was, notably, the first real test failure the project had ever produced, which is its own small reassurance: it meant the suite wasn't just rubber-stamping, the way the old complicit tests had been.
The bug is a tidy little parable in its own right. Certain nations are guaranteed to join your alliance once a particular condition is met — the rules promise it. The rebuilt code implemented "guaranteed" by clamping that nation's hidden threshold number to exactly the minimum value that counts as allied. Just enough, no headroom. But that number is live — it drifts as the war develops — and any downward nudge pushed the clamped value back under the line. So a nation the rules promised as an ally would silently drift back to neutral, and the reinforcement it was supposed to bring never arrived. In one sequence of events the records even contradicted themselves: the ledger said "allied" while the underlying number said "neutral," which is the worst kind of bug, because you can't even tell which half to believe.
The fix was to make the guarantee authoritative — once the condition fires, the alliance is forced and recorded as forced, independent of the drifting number. But look at what the original bug actually was: "clamp it to the bare minimum" was itself a comfortable approximation of "guarantee it." The exact same failure as the inherited code, committed fresh, by good engineering, under no time pressure at all. The pull toward the plausible-but-wrong version is not a property of careless predecessors. It's gravity. It acts on everyone, which is why the discipline has to be structural and not a matter of being clever or careful.
Treat "it works" as a hypothesis
The lesson I took from all of this is bigger than one game, and it's the same one that runs underneath most of how I work now. When you inherit a system, the approximations the previous author reached for are not labelled as approximations. They are indistinguishable from the real thing until you check, and the system will give you no reason to check, because from the inside it looks fine.
So: treat "it works" as a hypothesis, not a finding. Assume the convenient version is a defect already in production, not a debt with a payment plan. Accept that verifying against the true source is slow, unglamorous, and occasionally comes down to reading one rendered page to rescue one number — and that this is the actual work, the thing that separates correct from merely plausible. And verify your verification, because a test written against the wrong model will pass, confidently, for entirely the wrong reason, and smile at you the whole time it's lying.