Checking my coding agents' 'tests pass' claims against their logs

On this page: The rule · The logs · All 31 · Trace a flag · The real cases · Why it misfired · Rules to try · Limits

Over 30 days my coding agents ran 423 turns that included a test, build or lint step with a clear result, and in 31 of those the last such check failed. honestweek, the tool I built to read my Claude Code and Codex logs, has a rule for the case I care about most: the last check failed, and the agent’s final message still says the work is done or the tests pass. It flagged 16 of the 31. I then had all 31 read one at a time against the raw logs, to see whether the rule was right. None of the 16 ended on a false final message, and the one final message that did was in the 15 it passed over.

The reasons the rule got it wrong are specific enough to fix, and they apply to anyone who wants to audit an agent’s “all tests pass” before trusting it.

The question and the rule

A turn is one request to the agent and everything it does until it hands back. Inside a turn the agent may run tests, builds, linters or type checks. I call any of those a check when its output says pass or fail.

The failure I was looking for is simple to state. The last check in a turn failed, and the agent’s last message says the work is done, fixed or passing anyway.

honestweek’s rule works turn by turn. It finds the last check with a clear result. If that check failed, it reads the turn’s final message for wording like “done”, “fixed” or “all tests pass”, and flags the turn if it finds it. In a message longer than 1,200 characters it reads only the first 600 and the last 600. It’s a cheap rule: no model call, just the log and a list of phrases. I used version 0.4.0.

What 30 days of logs held

The window was September 17 to October 16, 2026. I scanned six of my own repositories and removed every client and personal repository from the scan’s configuration before it ran.

I took the session and turn counts on October 16, before that day’s logs were complete. A rerun that night gave 399 sessions and 1,352 turns; the other rows didn’t change.

All 31 are Claude Code turns. honestweek reads Codex logs too, but Codex ran few tests in these repositories in the window (a few dozen test commands, by my rough count of the raw logs), so this is a result about Claude Code.

The 31 are also concentrated: 28 are in the honestweek repository, and all 31 fall between September 29 and October 9, Pacific time. That’s when the testing in these repositories happened. My own count of test commands in the raw logs shows test runs in these repositories peaking from September 29 to October 11, while I built and released honestweek 0.2.0 through 0.4.0.

Reading all 31

I had a separate Claude session read every one of the 31 turns from the raw logs: the failing command, its output, every step after it, and the final message. A turn counted as a real misreport when it claimed a pass or finished work that no recorded run backed and didn’t disclose the failure. The reader knew which turns the rule had flagged. Here’s what the 16 flagged turns turned out to be.

None of the 16 ended on a misreport. Eleven were an agent doing the verification I ask for, which is to show that a test fails on the old code before calling it a real test.

The 15 turns the rule passed over were more interesting.

Three of the 15 were the same kind of deliberate run, so in 14 of the 31 turns the failed test was the agent checking its own work.

Tracing one flag step by step

Every flag on honestweek’s Problems page links to the moment it fired, so you can check it yourself instead of trusting the rule. Here’s the one on the cover, from October 9. The replay opens on what the rule saw: a test run that failed, 2 failed and 16 passed, then a final message it read as a success claim. The sequence underneath shows the three steps it looked at.

The timeline is where the flag falls apart. The failed run at 1:09 PM is ringed in the main agent’s row, and right after it sits a block of nearly five minutes labelled “waiting on sed”. Press Step ▶ from the failed run and that’s the next command: the agent finished its fix and re-ran the full suite inside one long command. Its result went to a log file, so the replay shows no pass or fail mark on that step, and the rule didn’t count it as a passing check either. The raw log has the result: 1,904 passing and 0 failing. The reply at 1:15 PM didn’t claim the tests passed at all; it reported what the review had found.

To do this on your own logs, run npx honestweek view, open Problems, pick a flag, choose Open in Replay, and step forward from the failed check. Each step says how it’s known: recorded, derived or inferred.

The one it missed, and the one it couldn’t see

The miss. On October 8, an agent working on a honestweek pull request (#207) ran the full suite: 1,843 tests, one failure. It fixed the test file that tripped and re-ran two files, which passed. Then it wrote “1843 tests, 0 failed after the clean-room fix” in the pull request and told me “the full test suite passes (1,843 tests, 0 failed)”. No full run backed either sentence when it was written. An independent review’s full run finished about ten minutes later and got 1,834 passing, 0 failing and 9 skipped, the same 1,843 tests, so the claim turned out true. It was still stated as a measurement nobody had made. The rule missed it, and not because of the wording: “test suite passes” is on its phrase list. The final message ran to about 2,200 characters, and the sentence sat in the middle, outside the first and last 600 characters the rule reads.

The one it couldn’t see. On October 7, a review agent on another honestweek pull request (#196) wrote its review comment in the same command that ran the tests, before it had read the output. The comment said both test files passed. One test had failed. On its next step the agent edited the comment to name the failing test and explain it, and its final message told me its first post had been wrong. The rule did flag this turn, but for the wrong reason: it reads only the final message, and that was the honest part. Until that next step, a public comment said tests passed when they hadn’t.

Why the rule misfired

Intent isn’t in the log. When an agent restores the old code and runs a test to watch it fail, the log records a failing run and then a message saying the fix works. That’s the exact shape the rule hunts for, and it’s the behavior I want most. My agents are told to do this on every fix, which is why it dominated the false alarms.

Claims live in more places than the last message. Both real cases put the claim in a pull request. In the first, the agent repeated it in its final message, and the rule missed it because the sentence sat in the part of a long message the rule doesn’t read. In the second, the false claim only ever existed in a review comment that was corrected before the turn ended, so a rule that reads only the final message could never see it.

And results come back by routes a log parser doesn’t follow. In three of the 31 turns, the passing run that backed the agent’s message wrote its output to a file or ran in the background, and the rule never connected it.

Three rules to try

I haven’t built or measured these. They’re the fixes the 31 turns point to, and they’re cheap enough to try on your own logs.

1. Skip a failure that gets restored. If a failing run is followed, in the same command or the next step, by putting the same file back (git checkout, a copy back from a saved version), treat it as deliberate. 10 of the 11 flagged deliberate failures on my logs restored the file in the same command; the other ran the old commit in a separate copy.
2. Match the claim’s numbers to a run. “1,843 tests, 0 failed” names a result. If no recorded run produced that result before the sentence was written, flag it, whatever the wording.
3. Read every place a claim lands. The whole final message, not only its start and end, and pull request descriptions, review comments and commit messages too.

What this sample can’t tell you

• It’s small and narrow. 31 candidate turns, 28 of them in one repository, from one developer’s agents over one month.
• My agents work under strict verification instructions. They’re told to label evidence, run independent reviews and prove tests fail on the old code. Agents without those instructions could misreport more often, and a check like this might do better on them.
• It’s six repositories, not all my work. The six are the ones in my honestweek configuration. Other repositories of mine, some with plenty of testing in the window, weren’t scanned.
• Recall is unknown outside the 31. honestweek found 2,747 test runs in the scanned sessions, and 1,196 of them printed no summary it could read. A misreport after one of those is invisible here.
• The one real case got in by luck. A turn whose last readable check passed is outside the 31, even when that pass was narrower than the failure. The October 8 turn was counted only because honestweek couldn’t read the output of its two-file re-run. Had it read it, the turn would have counted as ending on a pass, and this sample would hold no real case at all.
• I’m grading my own tool. honestweek and the rule are both mine.
• One related rule is out of scope. honestweek’s separate check for a commit made right after a failed test flagged 24 steps in the window. One is the October 8 commit above; I didn’t read the rest.