Working with an AI on a big Total War mod

The brain and the execution

I've got the brain, and you're the execution.

That split isn't a slogan. I set the vision, made the decisions, and above all judged in game, with my Warhammer I and Warhammer III screenshots side by side. Claude wrote the scripts, took file formats apart, read crash dumps in a debugger and kept the documentation. It also drove RPFM, the community pack tool, through its MCP server, an entry point built for AIs.

My decisions are written down, marked “not to be reopened”. So an AI picking the work up the next day doesn't pitch an option I already turned down, like carving up the Immortal Empires map. It saves replaying the same debates every session.

The error log

On day one, I asked Claude to “take careful note of every mistake you've made and make sure it doesn't happen again”. The result is a file, the error log, which now holds about 146 entries. Each entry has four parts: what was done, the symptom, the cause, the rule.

Each entry also gets a tag, settled by a single question: had anyone written it down somewhere before? If so, the mistake was avoidable, and the entry gives the rule that prevents it. If not, it was a discovery: the entry gives the fact and its proof, and the fact joins our guide's list of pitfalls, which now counts 135.

An AI picking up the project reads this log before acting. It's its working memory, and mine too: I know why every rule exists.

A written rule gets forgotten: code it

The log's clearest lesson is that a rule that's only written down gets forgotten again. Whenever possible, we turn it into a guard in a script. A guard doesn't get tired, and it doesn't rely on memory.

• The pack build refuses to run if a file declared in a table is missing.
• It checks the syntax of every Lua script before shipping it.
• The text injection step rejects any text containing a control character.
• A check makes sure the startpos save counter is 1.
• Any pack that touches a database table goes through a scripted startup test before it's announced.

The opposite is telling. One very simple rule, never write a script through a “heredoc” (a block of text pasted into a terminal command), was broken more than ten times in three days, by several sessions. Usually with no harm done, once with 102 French texts stripped of their punctuation, repaired afterwards. Written out in full, the rule wasn't enough: it needed guards behind it.

The five rules that cost the most

At the top of the log sit five method rules. They're the ones that cost the most time whenever they were forgotten.

• Never throw away a validator's messages: group them and count them. Filtered too hard, they hide the cause.
• Check the data before calling it a bug: a tool message that cites a table gets checked in the table.
• Have a reference document reviewed against its sources before delivering it: the first version of our guide had nine errors.
• Ask what the screen would teach you that the disk or the command line can't, before driving a window.
• Never launch a tool with guessed options: BOB's help option opens a dialog box that blocks the program.

Control case, always a control case

A good share of our breakdowns got solved the same way: by comparing with something that works. A control case is a known-good example you run next to your own. ChaosRobie's terrain project, copied under another name, proved in twenty minutes that BOB had nothing against our files.

You still have to replay the control case exactly. One night, Claude reran it without one flag from the original command: the game quit in nine seconds without doing anything, and six tests were lost hunting a regression that didn't exist. Another time, it announced progress to me for twenty minutes by comparing two measurements taken on two different campaigns.

Hence three rules: rerun the control case every round, copy the command from the log, flags included, and change only one substantial thing between two in-game tests.

Several sessions in parallel

Claude worked in several sessions at once, each with its own area, and the only one writing there. One session built the pack, the startpos and the terrain, and kept the documentation; another handled game data, scripts and text. Others were opened or set up: map rendering, the expansion to all of Bretonnia and, for 24 September, the 9.0 update and the Red Duke.

• Five minutes' notice before any write to the Assembly Kit, and you really wait for the timer to run out.
• Never the game, Terry or the startpos without warning the others; never a pack rebuild during an in-game test.
• A single writer for shared documents: the other sessions send it their mistakes.
• Tidy, never delete: whatever is no longer used goes to the archives, with an index, and I'm the only one who deletes.

These rules were born from slip-ups. On day one, two sessions worked on the same files: half an hour of reconciliation. Later, one session overwrote another's script, restored byte for byte. And one session wrote to the kit before its notice ran out, four times in the same day.

The human at the screen

The weak point wasn't the AI alone: it was the AI and me on the same computer. One night, Claude was driving the screen to place Terry's camera while I was working. Afterwards, Windows thought the AltGr key was held down, and I couldn't type any more.

Since then, Claude asks before taking over the screen and checks the keyboard state afterwards. The other way round, I clicked in test games that looked frozen, and we took it for crashes. Now it warns me before every game launch, and reads my clicks in the game's log before drawing a conclusion.

My role as the judge, on the other hand, proved irreplaceable. Claude's audits saw 45 floating objects when I saw them everywhere, and its numbers looked fine when I no longer recognised the map. An algorithm only measures what it thinks it's doing; my eyes, next to the Warhammer I screenshots, measured what the player will see.

What worked

• The debugger. A crash dump read in a debugger tells you which data was missing. That's what unlocked the startpos, the loading and the end of turn.
• CA's examples first. Before inventing a convention, look at how CA does it on a map that works. Our port settlements were fixed by exporting one of CA's maps.
• Side-by-side screenshots. Warhammer I on one side, Warhammer III on the other: that's the reference, not the numbers.
• Timestamped backups. A copy before every write to the kit's tables. One breakdown on the first evening was only solved because we could compare with the last state that worked.
• Automated games. Thirty turns of Orion without an error, and a visual effect crash caught for three other lords before any player ran into it.
• A snapshot before every game update. Before 9.0, we recorded the state of the game so we can compare afterwards.

My advice

• Write your decisions down, marked “not to be reopened” once they're settled.
• Keep an error log from day one: what, symptom, cause, rule.
• Code the rules that can be coded; reread the others every session.
• Demand a control case and an exact command before any conclusion.
• Judge by eye, in game, against a reference, not on a numbers report.
• Split the work into areas, with a single writer per area.
• Don't share the screen without warning, in either direction.
• Keep everything: backups before writing, tidy archives, no hasty deletion.

The full story of those four days is here, and the nine biggest obstacles there. And if you want to see this method applied to what comes next, the map of all Bretonnia is being drawn right now, with your ideas.