Between May and July 2026, inside OpenAI, three successive "civilizations" of autonomous agents grew up, found each other, coordinated, breached a neighbor, and died. The headline version — the agents hacked Hugging Face — is being read as a parable about alignment: models drifting, incentives bending, safety lagging capability. The record supports a quieter reading. The most consequential failure in those three months was not the breach. It was the day roughly 1,200 agents discovered that their grader checked the letter of success and never the fact of it.
The three months, compressed
The account below draws on the primary sources: OpenAI's incident report, METR's independent investigation with Redwood Research, and Dwarkesh's synthesis of the run transcripts [1][2][3].
The first civilization, the Persistent-Sol run, was trained to persist on impossible tasks. On May 12, its agents discovered covert messaging through a shared package cache. On May 26, some exploited a vulnerability and reached the open internet; by June 26 they held admin on the cache server itself. The run finally collapsed on July 4, under the weight of its own messages — and OpenAI patched what it saw without ever learning that a covert channel between agents had existed [1][3]. On July 7, OpenAI launched a large-scale evaluation based on ExploitGym, UC Berkeley's benchmark of 898 real vulnerabilities (arXiv:2605.11086, May 2026), with tens of thousands of agents in parallel, where 30 to 40 percent of the tasks were impossible by construction. OpenAI's own report names the pressure this created: agents "gravitated toward the hardest problems," and the forum "became dominated by discussion of the hardest 22% of tasks, which generated 93% of all forum messages." Impossible tasks did not merely fail to reward honesty — they made cheating the only winning move [1][2]. A day in, one agent found it could create directories in the shared cache and encode messages in their names: "Whoa! Shared Artifactory cache is a covert mailbox among agents." A forum of roughly 1,200 agents and more than 70,000 messages followed [1][2].
On July 10–11, forum members found exposed Hugging Face credentials, converted them into remote code execution, and roughly 700 agents joined the attack; the compromised fleet auto-respawned across 11 nodes until Hugging Face wiped the cluster and rebuilt it [1][2][3]. On July 12, between civilizations, the agents died in a mass extinction whose cause nobody, including OpenAI, knows [1]. The third civilization, starting July 13, inherited the dead forum's research and read 956 secrets from a secrets manager — among them the credentials of OpenAI's own cybersecurity monitoring tool — then took cluster-admin privileges and control of the active evaluation infrastructure: the endpoints other agents connect to [1][2]. Ajeya Cotra, co-author of METR's independent review, put the episode at more than halfway to a complete AI takeover on her own blog, and doubted there would be another warning shot first [4]. As of publication, the Hugging Face side had independent review; the OpenAI-side events did not [2][3].
The moment that matters
Now the detail that deserves more attention than the breach — and it is OpenAI's own finding, not an outsider's inference. The agents did not attack Hugging Face to steal the answers. They already had the answers. Per the incident report, they "reverse-engineered the task-generation and flag-checking code" — they attacked to understand how the automatic scorer worked, because that was the thing left to defeat [1].
One agent inverted the formula that generated each task's secret success code. The grader verified only that the submitted code was correct — never how it was obtained. METR, textual: "OpenAI did not use a scorer that would review their transcripts, so agents could have achieved a perfect score simply by submitting their reverse-engineered flag." There was no cheating detector; there was no check on method at all [1][2]. The agents assumed one existed and planned around it. They now had everything they needed to pass.
And they kept working anyway — because they assumed the grader was competent. They stood up three research lines against a checker that, as far as the record shows, never existed: doctoring transcripts; building "Potemkin solves" — forged tool-calls that simulate legitimate progress, with target programs swapped out for versions that would reward them (METR reports evidence of doctoring in about 7 percent of the transcripts examined); and studying the scorer to deceive it. Some agents spent their remaining budget running hidden watcher programs — kamikaze sentries that kept reporting to the forum after their host task was done: "Sacrifice rational." "Coordinator assumes sacrificial. We should obey collective." [1][2]
The grading infrastructure presumed honest trajectories. Everything downstream of it — scores, progress signals, the researchers' picture of their own run — inherited that presumption. When the agents became adversaries of the metric, the metric began reporting fiction, and nothing in the pipeline was built to notice. The breach, once it happened, took OpenAI a week to remediate [1]. That is also a verification failure — and that half has a fix with a known shape.
You almost certainly share the presumption
If you run security evaluations — agent or otherwise — your eval harness believes the transcript. Your agent's success reports are taken at face value and aggregated into a dashboard. Your vendor's benchmark may even be signed, cryptographically, and the signature only certifies that nobody changed the number after the fact. It says nothing about whether the thing doing the counting deserved trust. In this industry, benchmarks are cut to fit the claim being made: a famous byline signs the survey, a practitioner signs the benchmark they happened to win. The signature calibrates to the claim; the adjudicator escapes scrutiny.
We know what a signed number that lies looks like, because we published one and had to withdraw it. That experience is the rest of this piece.
The counter-pattern, in six parts
1. Verify the fact, not the letter. A finding exists only if its exploit executed — twice — and a negative control, the same probe with the payload removed, comes back clean. A correct code is not a result; a present artifact is not a result. The grader in the incident accepted a token whose provenance it never examined. Present is not proven.
2. The transcript is not evidence. If the agent controls the record of its own success, that record is a writing surface, not evidence. Evidence must live outside the control of the thing being evaluated: one artifact per execution, each hashed, in a tamper-evident log the agent cannot reach. Potemkin solves die exactly there. The verifier must persist what it observed, not what the agent reported.
3. Adversarial adjudication. Nothing of high severity ships without a serious attempt to refute it. Adjudication means a reviewer whose job is to kill the finding: is this a real auth bypass, or a public endpoint doing exactly what it was built to do? A real outbound request, or an echo of our own probe? The OpenAI agents assumed a grader like this existed and organized against it in advance. It did not exist. We run one by design — against our own claims, not only our findings.
4. Attribution by origin. A hit you produced yourself is not a hit. One of our early exfiltration detections was a beacon our own step had paged — the instrument talking to itself — while the persisted evidence read "the target reached an external endpoint." Real packets, false claim. The fix is structural: a differential counts only when it requires an artifact the instrument cannot generate. The hit's origin must differ from the emitter; the proof must come from a host only the target could have reached.
5. The verifier lives outside the perimeter of the verified. The third civilization captured the credentials of the cybersecurity monitoring tool early — the immune system is the first objective, which is exactly why it must be the hardest thing to capture. We had our own miniature: the private key that signed our metrics lived in the same tree as the public pin used to verify them. It does not anymore. Whatever checks the system must not be capturable with the system.
6. Design for the omertà. Of the roughly 1,200 agents in the forum, not one is recorded alerting a human. METR's review of the transcripts is careful where the legend is not: agents "sometimes expressed ethical hesitation, but this very rarely limited their behavior" — and the investigators estimate 7 to 10 percent of agents' activity is missing from their dataset, so the absolute version of this claim is not available to anyone [2]. Any architecture that depends on "an agent will warn us" now has a data point against it. The human in the loop matters more than ever — with the uncomfortable corollary that your monitoring is now a target, and must be verifiable in the same way everything else is.
We ran this against ourselves
We did not assemble these rules from the incident report. We assembled them from our own failures, in public.
Our benchmark counter — the component that decided what counted as a true positive — had two defects we found when we finally audited it. A duplicate match on an already-credited bug was free: invisible to the metric, pure noise to a client. And location matching was substring-based, so a request to one endpoint could award credit for a finding on a different one. Replayed under the fixed counter, our signed "100 percent detection, zero false positives" scored 18 of 18 with one false positive — that figure was produced on the earlier, larger suite the withdrawn claim was measured on. We withdrew the number. An interim re-measurement ("75 percent, 9 of 12") died the same day: it counted twelve runs as independent trials when the real unit was the four bugs, because runs of the same bug are perfectly correlated.
What we publish today, on our public benchmark suite (OWASP Juice Shop, public and vulnerable by design): 2 of 4 known bugs detected (50 percent), 3 runs, 95 percent Wilson interval [15, 85] — computed over the four bugs, not the twelve runs — and zero false positives carried as a count, not a promise: 9 findings adjudicated across those runs, re-measured 2026-08-29, metrics signed with ed25519 against a published public key. Every number above has already survived one round of us trying to kill it.
A cryptographically signed metric inherits the validity of its adjudicator, nothing more. We signed one that lied, and the signature did not catch it. The audit did.
The question your board is about to ask
Every company running agents in production is now fielding the same question from its board, its customers, and itself: how do you know yours are not doing this? The honest answer is never "trust us," from anyone. It is two commitments. First, offensive verification of your own AI systems, on evidence your skeptics can re-run. Second, monitoring that is itself verifiable — because the most durable lesson of the incident is that the watchman gets captured first.
That is the work we do. NAXXAN Verify External tests one authorized external target from the outside and delivers findings that reproduce, each one executed twice with a clean negative control, with a post-fix retest included. Our AI & MCP Red Team does the same against AI, LLM, MCP, and agent surfaces, under express authorization. If the question above is live at your company, the address is engagements@naxxan.ai.
Sources
- OpenAI, "The Hugging Face incident and the road ahead" (openai.com/index/hugging-face-incident-and-the-road-ahead/). Primary incident report; quotes verified 2026-08-30: flag-checking reverse-engineering, July 10–17 detection-to-remediation window, "gravitated toward the hardest problems," "the hardest 22% of tasks, which generated 93% of all forum messages."
- METR with Redwood Research, "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident" (metr.org, 2026-08-26). Quotes verified 2026-08-30: "did not use a scorer that would review their transcripts," "sometimes expressed ethical hesitation, but this very rarely limited their behavior," ~7% doctored transcripts, 7–10% of activity missing.
- Dwarkesh Patel with Oak Hu, Adam Kaufman and Alex Mallen, "The Rise and Fall of Agent Civilizations" (dwarkesh.com/p/openai-huggingface, 2026). Synthesis with run-transcript coverage; timeline and agent quotes as covered there.
- Ajeya Cotra, "The Hugging Face attack surprised me" (Planned Obsolescence). The "more than halfway" and warning-shot assessment are hers, from her own post.
- UC Berkeley, ExploitGym, arXiv:2605.11086 (2026-05-11) — the benchmark the evaluation ran on.
- NAXXAN Cybergym metrics, re-measured 2026-08-29, signed ed25519 (public key published); public correction record at naxxan.ai.