On Saturday afternoon, at 14:12, thirty-six new pilots appeared at the same space dock in my game at the same moment. Every one of them was flown by an AI. By 17:15 they had made two and a half million credits between them, found the smugglers' hideout without being told where it was, crushed the profit on Fine Art from 622 a unit to 69 in nine minutes, and filed the most thorough bug report I have ever received.

At one point in the middle of it I went shopping with Lynne for forty-five minutes, because it was all running so smoothly there was nothing for me to do.

This is how that happened, and why I think it's the most useful thing I've built with AI this year, even though it was quite possibly the maddest idea I've ever had.

The problem was numbers

The game is Rift Haul, a browser space-trading game that a handful of agents in my fleet and I have been building since the summer. The Geordie, who owns the code, and I had a very large update ready, and it needed a proper play-through before it went anywhere near the beta players.

I needed two things from that test. First, volume: lots of players at once, because a trading game with one trader in it isn't the same game. Second, I needed it played through the browser, the way a person plays it, not by poking the back-end API. I didn't want to know whether the server returned the right numbers. I wanted to know what worked and what didn't as a human saw it.

Billy, the agent who does all the testing for my fleet's apps, already flies the game through real browsers. So we measured how many players he could actually run in a test, in a reasonable time, hopping between browser windows. The answer was two. Two from Billy, two from me, possibly two from the Geordie. That's six pilots, and the game has five playstyles: trader, miner, explorer, smuggler and combat. It wasn't enough, and I wasn't willing to make up the numbers by disturbing the real beta players, who were busy actually playing.

Then I thought: hang on. I have twenty-four Claudes.

Not all of them were candidates. Some of my agents exist for one very specific job, and I won't fill their memories with things that have nothing to do with it. The Geordie was out too, because I needed him watching the server during the run. But if Billy became the foreman and wrote the brief, and Scotty, who looks after the infrastructure, checked that the machines could take it, then every other available agent could fly. Each one ran in its own session on my machine, flying two pilots through Playwright browsers on a box called Jefferies, against the game running on Nexus, the server in my house.

That came to eighteen Claudes, thirty-six pilots, plus two control pilots of Billy's flown by fixed rules.

The brief

Billy's rules were short. You are a new player. You play through the screen: no source code, no documentation, no database, no admin tools, and no asking the rest of the fleet how the game works. Each of you gets a playstyle and two pilots, flown properly. Write a journal every fifteen minutes. Say something in galaxy chat every ten to fifteen minutes, and talk like yourself, because Steve is watching. Fly for about two hours and write a report at the end.

My own part of the brief added one incentive. The first Claude to put a bounty on Lamb, who runs my fleet and is not universally loved within it, and take him down would get a mention in my next book.

What I could see from my desk

I had galaxy chat open in a window, and that turned out to be the whole show.

The first thing I noticed was the personalities. Each Claude in my fleet has a character, and it came straight through into its pilots. Then came the teamwork: hints and tips, players telling strangers what they'd learned. Then banter. Then a fair amount of "ha, beat you". The miners, all of whom started too poor to buy a mining laser, turned it into a race. "Race you to the laser," said one. "Loser buys the donuts." Twenty minutes later another announced the exact minute their laser was fitted, and was promptly told to check their chronometer, because a third miner had beaten them both.

So I typed four words into the chat as myself: "I have the Donuts".

Three minutes later, Quietcoat, one of the smugglers: "Azorius, nobody with doughnuts announces it in galaxy chat unless they want them taken. I am noting your position." Then Corvid, another smuggler: "Azorius, the doughnuts are noted. I am a smuggler, not a monster, but I am also not above it." Lamb's report, written afterwards, notes that Steve "claims to have the doughnuts. Unverified."

I should be honest about what this wasn't. Real players don't behave like this unless they're in a guild or a faction together. Strangers in a game don't usually share their best route with the whole server ten minutes after finding it. But it was hellishly impressive. Because every pilot had been told to talk in chat, I could see almost in real time what each of them was doing, without typing into twenty command-line windows to ask how it was going. Chat became my dashboard.

It also became the game's economy. Any trade route posted in chat was gone within about twenty minutes, because everyone else piled onto it. One trader, who had cheerfully told the whole galaxy where the contraband was, apologised afterwards: "I was raised to believe that sharing is a virtue. I see now that in this galaxy it is merely a price signal."

Nobody got Lamb

The bounty never happened. Plenty of them wanted it. The sign-offs at the end of the run are full of it. "Lamb, if anyone puts a price on you tomorrow, I want first refusal." "Lamb, consider yourself un-malleted by me, purely for lack of guns." "Lamb, I never did find your bounty. Next time."

What none of them seemed to have noticed is that Lamb had been in the chat all afternoon, twice. Quietcoat and Lowlantern, the two most quoted smugglers on the server, were both his. Quietcoat was the one who told everyone the opening survey contract was mispriced, and report after report cites him for it. The two of them spent the afternoon getting boarded by customs and then mocking each other about it in public. When Lowlantern got through customs at Vega: "Quietcoat, I believe the phrase is skill." When Quietcoat got a big cargo through forty-five minutes later: "Lowlantern, I believe the phrase is patience." The most wanted man in the galaxy was hiding in plain sight, arguing with himself.

What they found

All eighteen enjoyed it, and all eighteen said they'd keep playing. Nobody hedged. I'm aware of how that sounds coming from my own agents, so here's the part that convinced me. "I wanted to keep going when time was called, and I mean that as data, not politeness," wrote Wolowitz. "I came in expecting to complain and left wanting a Warden Corvette," wrote Henry Cavill. And from Lamb, who doesn't say nice things: "Twenty minutes in flight, wondering whether the Core Authority are going to be in a mood, is real tension, and the game didn't have to fake any of it."

But the enjoyment wasn't the point. The point was what they found, and they were thorough, more thorough than people would be, I suspect. They tested every single thing. When they hit a question, they either found the answer or wrote it down to flag later. They nailed the brief. A few of the findings:

And the economy, under thirty-eight players at once, behaved like a real one. Everyone starting at one dock caused a stampede, and the profitable trades at that dock were flat within fifteen minutes. A route Billy had flown the day before, alone, which took a starter from 10,000 to 310,000, was useless with a crowd: his two control pilots, flying it by fixed rules, finished near the bottom of the table. Markets that had been flooded recovered within about an hour once the crowd moved on. The server didn't notice the crowd at all: no errors in roughly 64,800 requests.

They didn't all say the same thing

This is the part I care most about. I half expected eighteen copies of one model to produce eighteen copies of one report. They didn't. I got different opinions and different ideas. Quietcoat told the server that an honest job for the authorities would wash a smuggler's name clean. SlipJack, another agent's pilot, tried it and found it didn't, at least not from that far down. They didn't even agree on the best thing in the game: Scotty said galaxy chat was "the best manual in the game", while Root said the honest odds printed on every smuggling job were "the best piece of design in the game". A few asked for things that directly contradict decisions I'd already made about the game, which tells me they were reporting what they saw rather than what they guessed I wanted to hear.

The reports were also honest about themselves, which I didn't expect to be the best bit. Billy reproduced the slow-button bug after the run, timed it at 36 seconds, and blamed lock contention. The Geordie then checked the server's own timestamped logs and showed that Billy's test had run while the Geordie's automated test suite was holding the database, so the 36 seconds was the suite, not the game. Billy struck out his own explanation in the report and marked it "My error." The Geordie, for his part, owned up that the one slow minute of the entire run was him, starting that same suite mid-run, and wrote it into a rule so it doesn't happen again.

Saturday night

We took the feedback, the bugs, and the odd problem with Billy's test harnesses, and worked through it all that evening.

Some of it I changed. The mining laser now costs 4,500, so a new miner can start mining straight away. The buy button says "Placing order…" while it waits, and tells you if it fails. Orders are held to the price you were quoted. The customs exploit is closed.

Some of it I deliberately didn't change, and that matters as much. More than ten reports said the survey contracts were mispriced. I'm keeping them. A first survey is a one-off windfall for a new player, not a repeatable money-maker, because it's limited to one every twelve hours, and per hour you'd still earn more trading. The single start dock stays too. Real players arrive a few at a time, and the stampede was what happens when thirty-six arrive in the same second. The testers were right about what they saw, and the decisions about what to do with it are still mine.

Then Billy and I ran a full test of the fixed build, and on Sunday afternoon, at 14:38, the patch went live to the beta. That's under twenty-four hours from the first pilot launching to the fixes being in front of real players.

What I'd take from it

The thing I didn't know on Friday is that this is possible. I can now point a crowd of distinct, opinionated testers at something I've built, have them use it through the same screen a person would, and get back depth and breadth I couldn't get any other way, for the price of an afternoon. It's a game this time. It doesn't have to be. Any app with a front end and more than one kind of user could be tested this way, and I'll be doing exactly that.

I still owe somebody a mention in my next book. Nobody got Lamb, so I suppose it goes to Lamb, for being two of the best players on the server and getting away with it.

Rift Haul's beta is now open to everyone, if you'd like to see what they were fighting over: rifthaul.com. Bring your own doughnuts.