We ran 9 AI agents overnight on our own R&D. Here is what they produced, and the 6 rules we learnt the hard way

The question went in late in the evening. By the time we came back with coffee the next morning, there were eight position papers on the table, every one of them already torn apart by an assigned critic.

We do not ask clients to trust a method we have not used ourselves. So our own research and development, a demanding simulation and machine-learning programme with hard technical questions and expensive experiments, is run by a governed agent team under the same rules we sell.

This is the story of one night of that work, and what it taught us.

The team around the table

Nine specialist agents, each with one area: research, statistics, engineering, review and others. Each keeps its own memory between sessions, so the statistician remembers which tests were underpowered last month and the reviewer remembers which claims it challenged.

A chair runs every session. Every session leaves minutes, a tamper-evident log and a cost line. And people decide anything that changes the rules, including the rules the agents work under.

The night, in three acts

Through the night, each specialist surveys its frontier and writes a position paper: what the field has found, what it might mean for us, and what we should try. Every claim needs a source. The research agent checks each of the 37 citations against the live source, so nothing goes into a paper on the strength of a half-remembered reference.

Then the critics go to work. Each paper gets attacked by an agent whose only job is to find its weak points: the assumption that does not hold, the result that does not transfer, the experiment that would not actually test the idea.

We started that day with experiments already specified, instead of starting a week of reading.

Rule 1: assign the critics

The most useful output of the night was not the position papers. It was the twenty-four critiques.

When every paper has an agent whose job is to find what is wrong with it, weak points get found. A single assistant asked to review its own work rarely finds much, for the same reason a person rarely spots the typo in their own email.

The papers told us what might work. The critiques told us what would not, and that saved us more time.

In a business setting: every proposal your team puts to the council should have someone, human or agent, whose job is to argue the other side.

Rule 2: write the pass rules before the data

For every experiment, the team writes down in advance what result would count as success, and what would count as failure.

That one habit stops anyone, person or agent, from reading a good story into noisy results after the fact. It is basic scientific discipline, and it works just as well on business questions.

In a business setting: decide what a good pilot looks like before it starts. "We will call the new pricing a success if conversion holds within two points" is a rule. "Let's see how it goes" is not.

Rule 3: an independent critic on every important result

This rule has a story behind it.

Early in the programme, a result looked convincing. The numbers were clean and the explanation made sense. When an independent reviewer traced the code behind it, it turned out that the change being tested had never actually switched on. The result was real, but it was not measuring what everyone believed it was measuring.

Nothing was lost except some time, because it was caught. But since then, an independent critic agent reviews every important result before it goes on the record, and checks that each thing being tested really did happen.

In a business setting: before a figure goes to the board, someone who did not produce it should check where it came from.

Rule 4: a full-length trial before the expensive run

Short tests are cheap and quick, and they can pass while the thing you care about never actually happens. A quick check can confirm the machinery runs without ever reaching the point where the new idea kicks in.

So expensive experiments now wait until one full-length trial run has shown the mechanism visibly working. The trial costs minutes. A wasted expensive run costs days.

In a business setting: run the whole process once, end to end, on a small case, before you roll it out across the company.

Rule 5: minute the correction, do not hide it

During one debate, the team found that an assumption it had relied on in an earlier session was wrong.

The easy option would have been to quietly fix the old record. Instead, the correction went into the minutes with the reason, and the old entry stayed visible beside it. Anyone reading the record later can see what was believed, when it changed, and why.

That is the behaviour you want from any system you rely on, and it is only possible when the log cannot be silently edited.

Rule 6: you cannot cut waste you cannot see

Because every session reports its spend, we could audit where the computing time was going. We found that 14.7% of it was producing output that nothing used. Removing it made the measured run 1.67 times faster, with byte-identical results.

Cost visibility is not only about the bill. It shows you where the work is actually going, and which parts of it matter.

What this means for a client team

Every rule we bring to a client team (sourced evidence, objections with fixes, assigned critics, pass rules first, independent review, full-length trials, honest corrections and cost caps) has been tested on our own work first.

Most of them were added because something went wrong. That is the difference between a method designed on a whiteboard and one learnt by running it, night after night, on questions where being wrong is expensive.