--- title: "The test every frontier lab reports" publication: deslop.media byline: deslop.media editorial provenance: editorial_synthesis provenance_note: "Editorial synthesis by deslop.media editorial, assembled from the subject's public record. The subject did not sit for an interview with deslop.media and has not reviewed it." issue: 1 issue_title: The Scaling Bet grade: draft issue_version: 3 issue_version_date: 2026-10-11 piece_version: 1 first_published: 2026-10-11 reporting_current_through: 2026-10 canonical_url: https://deslop.media/feature/francois-chollet/evaluation-as-governance/ cite_as: "\"The test every frontier lab reports,\" deslop.media editorial, Issue 1 (Draft v1, October 11, 2026)" changes: https://deslop.media/issues/1-the-scaling-bet/changes/ flag: "MCP tool flag_content, or POST https://api.deslop.media/api/flags" --- By December 2025, ARC Prize could say that ARC-AGI had been reported over the past year on model cards from all four major frontier labs: OpenAI, Anthropic, Google DeepMind and xAI. The foundation also tests frontier models itself and publishes the results, flattering or not. In May 2026 it reported OpenAI's GPT-5.5 at 0.43 percent and Anthropic's Opus 4.7 at 0.18 percent on ARC-AGI-3's Semi-Private set, a benchmark whose environments human testers had all solved. The pattern is the structural fact about François Chollet's institutional position. It is more important than Ndea's headcount, more important than the citation count on "On the Measure of Intelligence," more important than the prize purse on the latest ARC competition. All four major frontier labs have reported scores on a benchmark whose design and evolution rest with one foundation and, in practical terms, with one person. There is no other benchmark in current AI evaluation that has that property. The labs do not report because Chollet has commercial weight over them. He does not. Ndea, launched in January 2025, has no shipped product. The ARC Prize Foundation, which Chollet co-chairs and which became a nonprofit in January 2025, runs on a budget that is small by the standard of any of the four labs whose scores it tracks. The prize structure has grown — $1 million in 2024 across 1,430 teams; $2 million across three tracks in 2026, with the ARC-AGI-3 track alone at $850,000. None of that is the kind of pressure that would force a frontier lab to put ARC on a model card. The authority is the benchmark itself. ARC-AGI tests a property — sample-efficient generalization on novel compositional tasks — that the labs cannot easily capture by training-set construction or by post-hoc fine-tuning, and that the labs cannot easily replace with a benchmark of their own design without giving up the credibility of having reported on the existing one. Once a benchmark establishes itself as the operative currency for a specific kind of capability claim, it acquires a particular kind of authority: the authority of a test that a lab, by declining to report it, effectively concedes it cannot pass. ARC functions as a governance instrument rather than just a research benchmark. Frontier labs report ARC scores not because they have to but because not reporting would itself be a signal. Once all four labs have put ARC on model cards, a card that leaves it out risks being read as evasion by the audience the labs care about. The audience that matters here is not the public. It is the subset of customers, journalists, regulators, and other labs who track these comparisons closely. The useful comparison is with evaluation regimes in regulated industries, not with other AI benchmarks. Drug efficacy trials are designed by pharmaceutical companies but evaluated against criteria set by the FDA. Financial audits are produced by audit firms but conducted against accounting standards set by the FASB and IASB. In each case, the body that sets the standard is institutionally separate from the company whose product is evaluated. The arrangement is not perfect, but the separation is structural. In the case of ARC, the separations are weaker. Chollet authored the founding paper that defines what intelligence is for purposes of the benchmark. He co-chairs the foundation that runs the benchmark. He runs Ndea, an AGI lab whose program-synthesis approach is the kind of system ARC was built to detect. Each role is defensible on its own. He is qualified for each. The combination is unusual, and there is no parallel for it among the figures most often grouped with him in the AI evaluation discourse. Sam Altman's OpenAI and Dario Amodei's Anthropic report ARC scores, and neither operates the ARC Prize Foundation, though ARC Prize said in December 2024 that it would work with "the OpenAI team and others" to design its next benchmarks. Yann LeCun sets research agendas at AMI Labs, the startup he founded after leaving Meta at the end of 2025, but does not run the canonical benchmark by which his agenda will be judged. Chollet's defense, articulated in interviews and on X across 2025 and 2026, runs along the lines of: skill is not intelligence; ARC is a North Star for AGI research; the benchmark family is designed to give actionable goals for AGI research; transparency on scoring and competition rules limits the foundation's discretionary authority. Each of these is true in its own register. The competition rules are public. The scoring is reproducible. The foundation publishes its policy positions on its site, including the framing that ARC scores can "guide critical industry decisions." The phrase is the key one. The foundation describes ARC explicitly as a tool for industry decision-making, not just a research benchmark. The gap in the defense is the design process for each successive ARC variant. ARC-AGI-1 was published in a 2019 paper. ARC-AGI-2 was released in 2025. ARC-AGI-3 was previewed in 2025 and launched in March 2026. Each successor was designed in part around what the previous version had stopped discriminating on. There is no public adversarial process — no equivalent of the FASB exposure draft or the FDA advisory committee — by which a frontier lab, an academic researcher, or an outside critic could formally contest the design of the next variant before it ships. The cost of the concentration is not that the variants are wrong. The published structure of each ARC iteration has been defensible on the axis Chollet says they target. The process for verifying that defensibility runs entirely through one foundation and one co-chair. As long as the foundation's design choices keep being defensible after the fact, the arrangement holds. If at some point a variant is published that the broader research community judges to have moved off the original axis, the arrangement is harder to repair than a comparable failure in a more diffuse standards body would be. The ARC foundation will eventually have to engage this argument. It is not yet pressing because the benchmark variants have so far held to their stated axis and because the frontier labs have kept ARC on their model cards while the foundation publishes the low scores its own testing finds. The combination — credible benchmark, voluntary reporting by parties under no formal obligation — is what gives ARC its current authority. The combination is also, by construction, fragile. It would not take much to break it. A frontier lab that quietly stopped reporting ARC scores while continuing to report on benchmarks where it scored better would erode the convention. A foundation decision on ARC-AGI-4 that the frontier labs publicly contested as off-axis would erode the convention faster. Neither is on the record: ARC Prize's December 2025 account had ARC on all four labs' model cards, and no frontier lab has publicly contested a variant as off-axis. The foundation keeps shipping defensible variants. Ndea has still not posted a result, which limits the most aggressive form of conflict-of-interest critique because Ndea cannot yet be accused of having a benchmark designed to favor its own architecture. The structural arrangement — same person designs the test, runs the foundation that grades the test, and leads an AGI research lab whose approach the test was built to detect — is the live concern. Today it is held in equilibrium by the foundation's transparency, by Ndea's nonresult, and by the labs' reporting record. Each of those conditions can change, and the institutional structure as currently composed has no internal mechanism that would force a change in design behavior if any of them did. The scores do not settle the institutional question, including the September 2, 2026 result that moved ARC-AGI-3's 0.51 percent launch floor to 62.7 percent for GPT-6 Astra on the Semi-Private set under ARC Prize's Standard harness at max reasoning effort. The question is whether the labs keep reporting and the foundation keeps building defensible variants while one person holds the three roles that make the structure work.