NOT.02 · Notes · 20 October 2026
When model agreement is just a prompt echo
Why high agreement across model runs or research syntheses does not prove a classification is right — and how to design a test that catches shared systematic error.
In LLM-assisted classification work, it is easy to confuse reproducibility with accuracy. When three independent runs or two distinct model families assign an item to the exact same category, the natural instinct is to treat the decision as settled. In practice, unanimous agreement is sometimes the most dangerous failure mode of all: a systematic error baked into the prompt or the boundary rule itself.
Across recent projects I ran into this illusion in two distinct forms: first while building a concept graph from secondary sources, and again while running a blind reproducibility test on a taxonomy.
Trap one: consensus planted in the query
While building the skills intelligence concept graph, eight deep-research reports generated across two separate systems unanimously described the exact same five-stage organisational maturity ladder. At first glance, it looked like rock-solid domain consensus.
A provenance audit exposed the real cause: that very five-stage ladder had been pasted as an illustrative framing inside the prompt used to commission the reports. The models had not confirmed that the structure existed in the field — they had confirmed that they read the instructions and dutifully organised their output around them.
Agreement across ten sources that were handed the same leading question carries zero evidential weight. It measures obedience to the prompt, not the state of the domain.
From that point on, every extracted term in the pipeline passes through a provenance filter: concepts present in the commissioning prompt are isolated in a prompt-seeded register and cannot advance to an attested tier without an independent primary source that predates or stands outside our query.
Trap two: stable selection of the wrong leaf
A second variant of the same illusion surfaced during a blind test of a taxonomy of psychological games and phenomena (25 items, proper names masked, worked-example tables stripped, 3 independent runs per model across two model families).
On most items the models were both stable and accurate. On one borderline case, however, both model families — six runs out of six — picked the exact same leaf without hesitation. And in all six runs, that leaf was wrong against the intended architecture.
Had we measured quality solely by inter-annotator agreement across runs, that item would have scored 100% and sailed through review untouched. The defect became visible only because we had recorded the intended target leaf (ground truth) for each probe item before running the test.
What had gone wrong? An earlier gate in the cascade used an overly broad clause about “information asymmetry”. The rule was phrased so deterministically that any careful reader — human or model — was forced to turn left one gate too early. Run-to-run stability did not prove the classification was right; it proved the defect lived in the rule rather than in model noise.
Three rules for an honest taxonomy test
If a reproducibility test is meant to evaluate the structure rather than a model’s memory or compliance, it needs three constraints:
- Pre-recorded intent on a control set. Agreement between runs (stability) and agreement with the intended home (accuracy) must be tracked separately. High stability paired with low accuracy is the signature of a broken boundary rule.
- Stripping worked examples during testing. If a category boundary only works when accompanied by a list of named examples, taggers are matching by analogy to the examples because the rule itself is incomplete.
- Masking canonical names. If a test item is labelled “Prisoner’s Dilemma” or “Confirmation Bias”, a model will classify it from pre-training memory regardless of how your scope note is written. Masking the name forces the classification through your actual division criteria.