Our AI Visibility Monitor Counted 38 API Errors as Answers. Here Is How We Caught It.

Short version. In our 23 August measurement round, 38 of the 175 "answers" we collected from Claude were not answers. They were the text of an API error, 529 Overloaded, which the Claude Code CLI returned as a normal response with a clean exit code. Our judge scored each one as "brand not mentioned." We found the problem three days later while reading raw transcripts, re-issued the round with those records removed, and changed the collector so that server errors are recorded as errors. None of the round's conclusions changed. Two noise-band figures did, and one of them was exactly the kind of figure that would have mattered later.

We are publishing this because we sell AI visibility measurement. A measurement vendor that only publishes the rounds that went smoothly is asking you to take its numbers on faith.

What the symptom looked like

One cell of the round was wrong in two ways at once. In the problem-style question layer (six prompts where a buyer describes a pain without naming the category), Claude had triggered web search in 0 of 30 answers. Yet those same 30 answers contained 25 citations to status.claude.com. An engine that searched nothing does not cite a status page 25 times.

Opening the raw files explained it. The full text of each of those "answers" was:

API Error: 529 Overloaded. This is a server-side issue, usually temporary — try again in a moment. If it persists, check https://status.claude.com.

The CLI had printed the error where the answer should be, exited with code 0, and attached no error field. Our citation extractor then harvested the URL inside the error message as a citation.

Why our filter missed it

This was not our first contamination. In an earlier round, subscription-limit messages had come back as answers, so we added a filter for them. Its pattern matched "usage limit", "rate limit" and "session limit". "529 Overloaded" matched none of those, passed through as a valid sample, and was judged like any other response.

The general lesson is uncomfortable: every filter you write encodes the failures you have already seen. The defence is not a better regex. It is reading raw transcripts every round before trusting the aggregate.

How much it moved

Scope	Contaminated
All Claude records in the round	38 of 175 (21.7%)
Problem-style layer	25 of 30, leaving 5 real samples
Long-tail category layer	9
Brand-direct layer	4
Previous round, same collector	0 of 175

What did not change: no layer had crossed our pre-registered threshold before the correction, and none crossed it after. Claude's own-domain citation rate for us stayed at zero across every valid answer, which pointed to a retrieval-layer problem rather than a content problem, and that finding stands.

What did change: the noise bands. Our noise band is the round-to-round swing of control brands we never touch, and it sets the bar any later result has to clear. For the problem-style layer we had recorded a swing of 53.3 percentage points against the previous round, which would have made any future movement in that cell impossible to call significant. With the 25 fake records removed, that swing is 3.3 points, measured on the 5 real answers left. The 53.3 was an artifact and has been retired. The long-tail layer moved the other way, from 11.4 to 12.1 points, because the 9 records removed there had been dragging the control brands' rates down. Every later comparison uses the corrected round as its baseline.

If you check these figures against our lab page, one cell will look different. The lab page shows, for each cell, the largest swing between any two consecutive rounds, not only the latest pair. For the problem-style layer that is now 13.3 points, set between our two baseline rounds, so that is the number you will see there.

What we changed in the instrument

• The collector now recognises API Error: 5xx and Overloaded responses and stores them as error records, not samples.
• A single 529 does not stop the run, because it is a transient server condition. Three in a row do, the same rule we already applied to timeouts.
• The judging rubric was not touched. Changing the rubric mid-experiment would break comparability with earlier rounds, which is a worse outcome than the bug.
• The judge was not re-run. Error records have nothing to judge; the report was regenerated with them excluded.
• The original report is archived next to the corrected one, so the correction itself can be audited.
• The public numbers on our lab page were exported on 24 August, before the correction, and carried the contaminated figures until the export dated below. Worse, the page had used the 53-point swing as its showcase for why control brands matter, calling it engine behavior. The lab page now shows the corrected round, and that paragraph now says the swing was our bug.

Three questions to ask any AI visibility tool

1. What happens to a provider error? Is it dropped, stored as an error, or silently scored as "your brand was not mentioned"? The third answer inflates every "not visible" rate in the product.
2. What is n in each cell? A visibility rate whose denominator you cannot see is a rate you cannot trust. Ours dropped from 30 to 5 in one cell and the headline number would not have told you.
3. Can you re-judge from raw transcripts? If the tool keeps only scores, nobody can ever check them, including the vendor.

Disclosure

ClawWorld is an AEO (Answer Engine Optimization) service for B2B startups — AI agents that get your product mentioned, cited, and recommended by ChatGPT, Perplexity, Gemini, and Google AI Overviews, with a measure–execute–remeasure loop. The measurement described here is the one we run on ourselves. The corrected round is published with control groups and noise bands at claw-world.app/lab; the round we measured on 7 September will join it once a hand check of its judged answers passes. The method is written up at claw-world.app/methodology.

Corrected lab data exported: 2026-09-26.