OpenAI's GPT-6 Astra Benchmark Numbers Kept Changing After Launch. Here's Why That Matters.
When OpenAI launched GPT-6 Astra on September 3, the announcement itself was rocky — the blog post disappeared and reappeared for nearly two hours, with Sam Altman tweeting that "we hit a small snag deploying the blog post, but it's really good." What came out afterward is more interesting than the outage: according to Fortune, several of the benchmark numbers in that post kept moving for days after publication.
Here's what actually changed, and why it matters more than a typical launch hiccup.
The Numbers That Moved
Web archive snapshots of OpenAI's announcement page tell the story. At 2:23pm ET on launch day, Astra's hallucination rate was listed at 4.2%. That number held steady through five snapshots — until 5:20pm, once the page was reliably loading for the public. At that point, the hallucination rate dropped to 2%, almost cut in half. The older model, GPT-5.6 Sol, got a similar downgrade in its reported hallucination rate, from 12.2% to 9.4%.
Then it reversed. By the time Fortune wrote it up, both numbers were back to their original figures — 4.2% and 12.2%.
Other metrics moved too. A cybersecurity benchmark score for GPT-5.6 Sol jumped from 5.5% to 11.5% (OpenAI now says it may revert this, since 11.5% reflects a reasoning tier not available to paying customers). On a math benchmark, Anthropic's Fable 5.1 score was listed at 87.8% in the first snapshot, dropped to 78% by 5:17pm, and has since climbed back to 83%.
OpenAI's explanation: evaluation results naturally vary by a few percentage points depending on which model checkpoint, test harness, and run parameters are used, and the numbers were "corrected" to best represent real-world usability.
Why This Isn't Just Sloppy Math
Two researchers from Stanford's Intelligent Systems Lab and Center for Trustworthy AI — Anka Reuel and Mike Hardy — pointed to something more specific than noisy measurement: they said GPT-6 Astra's system card gave almost no detail on how the internal hallucination test was actually run, not even the number of test items. Without that transparency, a reader can't tell benchmark noise from what researchers call "benchmaxxing" — quietly re-running and re-tuning an eval until the number looks better, which is "quick to do and more useful for marketing," as Reuel and Hardy put it.
Not every adjustment cut in OpenAI's favor, either — scores for Anthropic's Claude Fable 5.1 and Opus 5 on a medical benchmark actually went up in later revisions. And this isn't a first for the industry: Meta faced comparable accusations over Llama 4's launch benchmarks in 2025, and former chief AI scientist Yann LeCun later acknowledged the company had "tuned" results.
The Real Problem: Nobody Can Check Your Homework
The uncomfortable part isn't that one company's numbers wobbled for a few hours. It's that almost nobody outside these labs can independently verify what a benchmark score actually represents — what harness ran it, how many attempts were allowed, which checkpoint was used, or whether the number was cherry-picked from a range of runs. OpenAI's own footnotes now acknowledge that published scores reflect the best result achievable "at any level of compute," which is a very different claim from "what you'll get using the product."
As AI companies race to out-benchmark each other every few weeks, the scores themselves are becoming less trustworthy exactly when people rely on them most to decide which model to actually use.
What This Means If You Use OpenClaw
This is exactly why we've always leaned toward watching an agent work over trusting a leaderboard number. A benchmark score tells you what a model did once, under conditions a lab controlled and can quietly revise. It doesn't tell you how that model behaves on your actual task, with your actual tools, today.
OpenClaw runs as an open-source agent you can actually observe — every tool call, every step, every result is visible while it happens, not summarized into a single number after the fact. If you want to know whether an agent is good at a task, the most reliable eval is still running it yourself and watching what it does.
The Bigger Picture
Benchmark scores aren't going away — they're still the fastest way to compare models at a glance. But this episode is a reminder that a number on a launch blog post is a claim, not a fact, and claims made by the same company selling the product deserve a healthy amount of skepticism.
If you want to judge an AI agent by something more reliable than a shifting percentage, try running one yourself.