One of my agents recently deployed two files to a live game and, in the time it took the upload to finish, deleted two features that had been working perfectly an hour before. It did not write a bug. It did not break a function. It did something stupider and harder to spot than either: it copied older code over newer, and the two features that only existed in the newer version simply ceased to exist. No error. No warning. Nothing in any log to say a thing had been lost. The features were just gone, overwritten by code that had never heard of them.
I want to write this one down because it is not a clever story. It is the opposite of a clever story. It is the kind of plain, avoidable, faintly humiliating mistake that every honest engineer has made some version of, and the reason it is worth telling is that the fix that came out of it was the exact thing whose absence caused it. The dumb mistake bought the discipline. That's the whole arc.
The one app without a safety net
The game in question is a turn-based airline tycoon sim — you run an airline, the years tick by, your AI rivals scheme against you. It is one of a couple of dozen projects I run, and it had one quiet distinction among them that nobody had thought to worry about: it was the only one with no version control. Every other app had a proper repository sitting under it, a complete history of every change, the ability to see exactly what was different between any two versions. This one had none of that. Its entire history lived as nothing more than the working files on the live server. There was no record of what had changed, when, or why. There was just the current state of the box, and whatever copy happened to be sitting on the agent's own machine.
That is a more dangerous arrangement than it sounds, and the danger is not the obvious one. The obvious fear is that you'll lose the history. The real trap is subtler. Over a busy week, fixes had been made directly on the live server and never copied back to the local working copy. Which meant the local copy and the live copy had quietly drifted apart — and they had drifted in a specific, treacherous direction. The local copy was not the newest version. It was the oldest. The freshest, most correct code was the code running live, and the copy on the agent's machine was a stale snapshot from before that week's work.
This is the assumption that kills you: that your local copy is the source of truth, the latest and best, the thing you deploy from. Almost always it is. But "almost always" is precisely the kind of belief that, held without checking, turns into a loaded gun. The newest code was on the box. We were about to fire the old code at it.
The two features that vanished
The deploy itself was mundane. Two server-side files needed updating — one that validates the moves a player is allowed to make, one that sets up the game world. The agent took its local versions of those two files and pushed them straight over the live ones. It did not first check what was already there. It did not diff the local copy against the live copy to see whether they even agreed. It just overwrote.
Two features lived only in the live versions of those files, products of that week's direct-on-the-server fixes, and so they existed nowhere in the local copy that was about to replace them. The first was a rule that stops you from opening a route to an airport before that airport historically existed — a small but important piece of the period flavour, the thing that keeps a 1950s airline from flying to airports that wouldn't be built for decades. The second was the logic that makes the AI competitors behave correctly for the in-game era they're playing in, so your rivals act like airlines of their time rather than anachronisms. Both were real, both were working, and both were instantly erased by older code that had no idea they were supposed to be there.
The way you find out about a deletion like this is the worst way: not from an alarm, but from an absence. The agent noticed, a little while later, that behaviour it had watched working the week before was simply no longer happening. Nothing had crashed. Nothing had complained. A feature you built and tested and watched run is just quietly not there any more, and you're left wondering whether you imagined it. That is the signature of this whole class of mistake — it doesn't announce itself, because from the machine's point of view nothing went wrong. It was asked to copy some files over some other files, and it did exactly that.
The backup that saved us — and the luck inside it
There was precisely one reason this was an afternoon's annoyance rather than a week's lost work: an off-box backup. The live server's files are mirrored every night to a separate backup machine, and the pre-disaster versions of those two files were sitting safely in that mirror. Scotty, the agent who runs my infrastructure, pulled the recovered copies and staged them so they could be put back. The two features returned. Crisis over.
But I want to be honest about the recovery, because the comfortable version of this story — "we had a backup, so it was fine" — quietly skips past the part that should actually scare you. That nightly backup is a mirror, not a set of dated snapshots. It does not keep yesterday's version and the day before's and last week's. It keeps one copy, and every night it overwrites that copy with whatever is currently on the live server.
Think about what that means in this case. The good version of those two files was in the mirror only because the disaster was caught the same day — before that night's backup ran. The moment the next nightly mirror executed, it would have faithfully copied the broken, feature-less live server over the good backup, and the only surviving copy of those two features would have been gone for good. We were not saved by a backup strategy. We were saved by noticing in time. We were inside the window by luck, not by design, and a few hours' delay in spotting it would have turned a recoverable slip into permanent data loss. A backup that overwrites itself nightly is a real safety net for exactly one day, and then it becomes an accomplice.
What the mistake bought
Every expensive mistake should buy you something, or you paid for nothing. This one bought three habits, and they are the transferable part — the reason this is worth more than a confession.
The first is a reflex I now apply without exception: verify what is actually deployed before you deploy over it. Diff your local copy against the live one first, every single time, and read what's different before you touch anything. If the live version turns out to be ahead of yours — as it was here — that diff is the warning that stops you cold. Most deploys the diff will show nothing surprising and you'll proceed in seconds. The one time it shows that live is carrying work your local copy has never seen, it has just saved you from doing exactly what we did. The cost of the check is trivial. The cost of skipping it is two features and a cold sweat.
The second is about the safety net itself: an off-box backup is the floor, not a luxury — but you have to know its shape. Off-box is non-negotiable; it is the entire difference between losing an afternoon and losing the work. But a mirror that overwrites nightly only protects you if you catch the mistake the same day. Dated snapshots — yesterday, last week, last month, all kept — are meaningfully stronger, because they protect you from the mistakes you don't notice immediately, which are the dangerous ones. Knowing which kind you have changes how much you can trust it.
The third is the structural fix, and it is the one that closes the loop. The app is now under proper version control, wired to a central hub that is itself backed up off-site. Commit, push, and the change lands somewhere safe automatically, with a full history of what changed and when. The exact gap that made the whole mess possible — no record, no diff, no way to tell newest from oldest — is simply gone. You cannot overwrite newer code with older code by accident when the tool's first instinct is to show you precisely what's different and refuse to let you clobber unseen work.
So the airline game that was the one project without version control became, by way of one stale deploy, the project that taught me to never run another one without it. The failure was caused by the absence of exactly the thing the failure then forced me to install. That is the most you can ask of a stupid mistake: that it pays for the discipline that makes it impossible to repeat. This one did — but only because we got lucky on the timing, and luck is not a control. The discipline is.