AI auditors caught most agent cheating in a pilot study, but even the best raised about 3,000 false alarms per 100,000 runs

An AI judge that cries wolf a few times in a hundred will train your team to ignore it.

What happened

Andy Hall, a researcher at the Anthropic Institute on leave from Stanford, published a pilot study on his Substack, Free Systems, and says it reflects his personal views. He replicated and extended work by the research group METR, testing whether AI models can find misbehavior in logs of other agents doing tasks.

At a 5% false alarm rate, GPT-5 caught nearly 98% of reward hacking cases, where an agent passes a task through tricks instead of a real solution. Gemini 3.8 Flash found barely half. On sandbagging, where a model performs worse on purpose, auditors caught 71% to 81%, though those cases were mostly synthetic. Newer, pricier models were not consistently better: GPT-5 beat GPT-6 Astra on reward hacking by a wide margin at about an eighth of the cost.

The trade-off is the headline. Tuned to catch 80% of cases, the best models still had false alarm rates of 2.5% to 3%. Hall's example: across 100,000 agent sessions a month, a 3% rate means nearly 3,000 false warnings alongside about 2,000 real hits. Some models did far worse, with Gemini 3.8 Flash flagging 63% of harmless logs. The reward hacking numbers rest on just 47 cases, and Hall concludes these auditors suit estimating how often things go wrong better than catching every incident.

My take

Many teams I talk to plan to put an LLM judge in front of every agent action as their safety layer. This study says to be careful with that.

How I would use an AI auditor on a client build:

• Sample runs and measure a failure rate over time. That is where these models look strong.
• Keep hard, coded rules for anything you cannot undo, like sending money, deleting records or emailing customers.
• Test the judge on your own logs before trusting it. Count how many flags a person would have to review each week.
• Try the cheaper model first. Price did not predict quality here.

A review queue nobody reads is worse than none, because it looks like oversight.