The ARC residual gap

There is a familiar criticism of François Chollet’s Abstraction and Reasoning Corpus, made often in public commentary since OpenAI’s o3 results: every time the frontier models start to score well on ARC-AGI, a harder version appears, and the goalpost moves. By that telling, ARC is unfalsifiable by construction. No system can ever clear it because the benchmark redefines clearance whenever clearance threatens.

The criticism gets the design exactly wrong. It also identifies something real.

Chollet introduced ARC in his 2019 paper “On the Measure of Intelligence,” not as a candidate AGI diploma but as an operational test of what he called “skill-acquisition efficiency.” The framing was explicitly negative. AI benchmarks up to that point had measured skill — performance on a fixed task — and Chollet’s argument was that skill could be acquired by enough training data on the specific task, which made performance on it a poor proxy for the underlying capacity to learn. The corpus was small visual puzzles, each requiring inductive generalization from a handful of examples to a held-out test case. ARC was, from the first paper, a measure of the gap, not a finish line.

ARC-AGI-1 saw little progress from frontier models for five years, though specialized systems pushed the private-evaluation score to 55.5 percent in the 2024 competition. Then on December 20, 2024, OpenAI announced o3-preview results of 75.7 percent on ARC-AGI-1’s Semi-Private evaluation at low compute and 87.5 percent at high compute. Chollet’s reaction on X that day was to call the result a “significant breakthrough” while explicitly saying it was not AGI. The breakthrough part is the one most quoted. The not-AGI part defines the design.

ARC-AGI-2, released in March 2025, ran the original conceit on a different distribution of puzzles. On its Semi-Private set, production o3 at medium reasoning scored 2.9 percent in April 2025. By December, Anthropic’s Opus 4.5 (Thinking, 64K) was at 37.6 percent, and a refinement system from Poetiq built on Gemini 3 Pro reached 54 percent. ARC-AGI-3, launched on March 25, 2026, reset the picture again. Human testers had solved every environment; frontier AI scored 0.51 percent at launch, on a measure weighted by how efficiently a system acts compared with human players. In May 2026 ARC Prize reported GPT-5.5 at 0.43 percent and Anthropic’s Opus 4.7 at 0.18 percent on the Semi-Private set.

Each ARC iteration is cleared, slowly, by frontier systems acquiring whatever specific capacity it tested for, and then a successor iteration tests for something the prior systems demonstrably cannot do. The intervals are getting shorter. Frontier models took roughly five years to reach high single digits on ARC-AGI-1, then cleared most of it within a quarter once reasoning models arrived. ARC-AGI-2 went from 2.9 percent in April 2025 to 54 percent that December, and ARC Prize’s September 2026 results put GPT-6 Astra at 95.0 percent on its Semi-Private set at max reasoning. On September 2, 2026, ARC Prize reported GPT-6 Astra scores of 62.7 percent on ARC-AGI-3’s Semi-Private set using the Standard harness at max reasoning effort, and 99.95 percent using the Provider Adapter harness at high reasoning effort.

This is the pattern critics call goalpost shifting; Gary Marcus’s often-cited use of the phrase, on December 20, 2024, was aimed at readers who said o3 refuted his own “wall” prediction, not at ARC. Chollet’s response, on X in March 2026, made the design explicit: “not a final exam,” “moving target,” “actionable goals for AGI research.” ARC-AGI is not a single test designed to be passed once and retired. It is a benchmark family designed to track the residual gap between human and machine learning by, when each iteration is solved, defining the next one around what is still unsolved.

Benchmarks that get solved and stay solved stop being useful. ImageNet is the canonical example: once frontier models cleared it, it became a regression test rather than a research signal, and the field had to build harder benchmarks behind it. The argument for a self-redesigning benchmark family is that the discipline of building it forces a clear articulation, each generation, of what specifically remains hard.

The ARC Prize Foundation, which became a nonprofit in January 2025 with Chollet as co-chair, frames the project in those terms. The prize structure rewards open-source progress on the current ARC variant: $1 million in 2024 across 1,430 teams; 1,455 teams and 90 papers in 2025; $2 million across three tracks in 2026 with the ARC-AGI-3 track alone at $850,000. The competition format is a partial answer to the goalpost criticism: the rules of the current benchmark are fixed long enough for a year of competitive submissions before the next variant arrives.

A benchmark designed to keep moving is, by construction, never finished. That is valid measurement if each successive iteration tests something genuinely harder along the same underlying axis — abstraction, novelty, sample efficiency. It is unfalsifiable framing if each successive iteration just gerrymanders around whatever the leading model has just learned to do. The two cases produce identical trajectories from the outside.

The cost is real, and Chollet does carry it. The decision about what counts as “genuinely harder along the same axis” rests with him and with the foundation. There is no public adversarial process by which a frontier lab could contest the design of the next ARC variant before it ships.

In defense of the design, the published structure of the ARC variants has been, so far, defensible on the axis Chollet says they target. ARC-AGI-1 tested compositional generalization from few examples on visual analogical puzzles. ARC-AGI-2 tightened the same conceit with harder puzzles that human testers could still solve. ARC-AGI-3 expanded into procedural and interactive tasks that human testers could solve and on which frontier systems scored 0.51 percent at launch — a gap large enough that the argument for moving the target had its own evidence, until ARC Prize reported GPT-6 Astra at 62.7 percent on the Semi-Private set under its Standard harness, at max reasoning effort, in September 2026.

If at some future generation a successor ARC turns out to be one that a non-trivial human population also struggles with, the human-AI gap will collapse not because models got better but because the benchmark moved off the axis it claimed to be on. Chollet’s framing — “skill is not intelligence,” “skill-acquisition efficiency,” “not a final exam,” “moving target,” “North Star for AGI” — provides the rhetorical defense; in September 2026 he added a bar for declaring AGI: AI capable of invention, including conceptual breakthroughs, new insights, and new technology. The empirical defense has to be reconstructed each ARC iteration, and the people doing the reconstruction are the same people running the benchmark.

The goalpost criticism, as usually stated, is wrong: the benchmark was never meant to be a finish line and Chollet has been clear about this since 2019. The sharper version is correct: a benchmark family that the same person designs, redesigns, and grades is more vulnerable to drift along its central axis than a benchmark fixed at a single point in time. The first version of ARC was a paper. The current version is an institution. Institutions are harder to keep honest, and ARC’s institutional architecture is more concentrated than the architecture of any other major AI benchmark in current use.

ARC is a working benchmark of the residual gap. Its self-redesigning structure is a feature of its measurement function. The design carries an irreducible cost in trust that the foundation has so far paid by being transparent about its scoring, public about its competition rules, and consistent in its framing. The cost is not zero. It scales with each iteration, because each iteration moves the question of what counts as harder one step further from the original 2019 paper.

Whether that cost is sustainable is a question about the foundation’s process, the prize’s competitive credibility, and the labs’ willingness to keep reporting scores even as the variants get harder. So far, all three have held; ARC Prize’s December 2025 account had ARC on the model cards of all four major labs. The arrangement is unusual. It is also, on the evidence to date, working.