I Ran My Own Analysis Past a Second Model Five Times. I Retracted Twice.

I had a piece of analysis I liked. It explained a strange result in my own analytics, it was internally consistent, and it made my work look good. So I handed it to a different AI model and asked that model to take it apart.

We went five rounds. I retracted twice. The second model conceded once. And the question that survived all five rounds got answered in about ninety seconds by a chart I had been scrolling past the entire time.

This post is about the method, because the method transfers. The specific finding is a separate post.

The setup

One model builds the analysis. A different model, in a separate session, receives the analysis as a document and is instructed to falsify it. I carry the messages between them and run any test either one proposes.

Two rules make this work rather than turning into two chatbots agreeing with each other.

Each side must name what would change its mind. Not “I disagree,” but “the following measurement would make me wrong.” A round that produces no falsifiable statement is a wasted round.

The messages are documents, not chat. Every number carries its instrument, its window, and the date it was read. Both models then argue about the same object rather than about their memories of it.

What actually happened, round by round

Round one. I sent my analysis with first-party data attached. Three of the other model’s premises were falsified by that data. It accepted all three without hedging. That was the first sign this was worth continuing, because a model that just agrees is useless and a model that just defends is worse.

Round two. It came back with a real objection: I had treated a small exported dataset as a complete census, and the vendor’s own documentation describes that export as a sample. It was right. I had built an argument on top of “there are only 26 of these” when the documentation said the number was a sample of an unknown larger population. Retraction one.

Round three. I queried the underlying report endpoint directly and it returned a total record count matching the export. So the small number was not a display truncation, and my retraction had been slightly too generous. Correcting your own correction is unglamorous and it is part of the method.

Round four. I found a new field in the data that appeared to explain everything at once, and I wrote it up as a mechanism. The other model refused it. Not the observation, the causal direction. The vendor documents that field as a way of grouping results after the fact, and warns that it can be imprecise for small or new categories. So my data was consistent with two very different worlds, and I could not tell you which one I was in. Retraction two.

Round five. I read the daily chart. The metric had been near zero for twelve consecutive days, jumped to a large number overnight, then held flat for eleven days with no weekly variation at all. Human behaviour does not have that shape. Something switched on. Every explanation both models had built across four rounds was an explanation of a rate, and the thing in front of us was not a rate.

The lesson I did not expect

Four rounds of sophisticated argument, two retractions, one direct API query, and a search through vendor documentation. The thing that reframed all of it was the default chart on the report’s landing page, which I had looked at and not read.

I had been treating a total as the finding and the shape as decoration. That is exactly backwards. A total tells you how much. A shape tells you what kind of process produced it, and what kind of process it is determines whether the total means anything at all.

Cheap instruments first, then. Before you commission the study, look at what is already on the screen. This is the same discipline I apply to generated code, where a build exit code outranks a confident summary: see how I verify AI-generated code before shipping.

Where two models beat one

Being wrong out loud, repeatedly, in front of something that would not let it go.

Working alone, I would have shipped round one. It was coherent, it was supported by real data, and it was flattering. The second model did not know more than I did about my own site. What it had was a job: find the load-bearing assumption and push on it. That is a role, not an intelligence advantage, and roles are cheap to assign. Which model fills the role matters less than the role itself, though it is not nothing: I wrote about how I route work between Fable 5 and Opus 5 elsewhere.

There is one failure mode worth naming. Two models will happily converge on a fluent, agreeable, wrong answer, because both are optimised to sound reasonable. The counter is the falsification rule. If neither side can state the measurement that would end the disagreement, the conversation has stopped being an investigation and has become prose.

Running it yourself

1. Write the analysis as a document, with every number carrying its instrument, its time window, and the date you read it.
2. Hand it to a second model with one instruction: find the load-bearing assumption and try to falsify it.
3. Carry the reply back yourself. Do not let either side summarise the other.
4. Every round, both sides state what evidence would change their position. No falsifiable statement means the round produced nothing.
5. When a proposed test is cheap, run it before arguing about it.
6. Keep the retractions in the document, dated. They are the part with the highest information content.

What I am left holding

Less than I started with, stated more precisely, plus a scheduled recheck two days out that will decide between three remaining explanations.

That is a worse story and a better position. The version I would have published after round one was more impressive and would have been wrong in a way that took months to surface, probably in front of someone who had paid me.