Moving-target discipline
ARC-AGI changes what kind of test it is rather than raising the bar on the old one. A version that raised the passing score without changing the task would prove the goalpost charge. None has.
The strongest critique of ARC-AGI as a benchmark family is not technical. It is structural: each new version appears just as the field’s strongest models are getting traction on the previous one. Gary Marcus’s “goal post shifting” post of December 20, 2024 was aimed elsewhere: at readers who said o3 refuted his “wall” prediction, which he said had not been about ARC. Emily Bender, the University of Washington linguist who co-authored “Stochastic Parrots,” and Melanie Mitchell, the Santa Fe Institute computer scientist, have written about how far benchmark scores can be read as evidence of general capability. Demis Hassabis’s DeepMind has prioritized publicly fixed targets (AlphaProof and AlphaGeometry on IMO problems, AlphaFold on CASP) that do not version on the same cadence. The charge, if it held, would be serious — because if ARC-AGI is moving, the falsification instrument that gives Chollet’s frame its empirical teeth is being preserved through evasion rather than rigor.
ARC-AGI-2 arrived three months after o3’s ARC-AGI-1 jump; ARC-AGI-3 followed a year later
ARC-AGI launches and frontier results, Nov 2019 to Sep 2026
- Nov 2019 ARC introduced Chollet publishes On the Measure of Intelligence (arXiv:1911.01547) and releases the original ARC dataset.
- Jun 2024 ARC Prize 2024 launched $1M open competition against ARC-AGI-1; 1,430 teams enter.
- Dec 2024 o3-preview clears ARC-AGI-1 OpenAI announces o3-preview at 75.7% (low compute) and 87.5% (high compute) on the Semi-Private set. Chollet calls it a “significant breakthrough.”
- Mar 2025 ARC-AGI-2 announced (same regime, harder) Same format, harder tasks; in development since 2022. Production o3 at medium reasoning scored 2.9% on the Semi-Private set in April 2025.
- Nov 2025 Opus 4.5 reaches 37.6% on ARC-AGI-2 Opus 4.5 (Thinking, 64K), Semi-Private set, Nov 24, 2025. Poetiq’s refinement system on Gemini 3 Pro reached 54% on Dec 4, 2025.
- Mar 2026 ARC-AGI-3 launched (qualitative regime shift) Launched at Y Combinator, March 25, 2026: interactive environments, scored on action efficiency against human players. Human testers solved every environment; frontier AI scored 0.51% at launch. In May 2026 ARC Prize reported GPT-5.5 at 0.43% and Opus 4.7 at 0.18% on the Semi-Private set.
- Sep 2026 GPT-6 Astra · 62.7% on ARC-AGI-3 ARC Prize reported 62.7% on the Semi-Private set under its Standard harness at max reasoning effort (Sep 2, 2026), and 99.95% under the separately scored Provider Adapter harness at high reasoning effort.
Sources: F. Chollet, “On the Measure of Intelligence”, arXiv, Nov 2019; ARC Prize, ARC-AGI-2 announcement, Mar 24, 2025; ARC-AGI-3 launch, Mar 25, 2026; GPT-6 Astra results, Sep 2, 2026
What “moving target” means in Chollet’s frame
Chollet’s defense, set out on X and in the launch material when ARC-AGI-3 opened in March 2026, is structural: the benchmark’s object is not a fixed score threshold. It is the gap. ARC-AGI-1 was the first attempt at isolating a kind of generalization that scaling does not solve. When o3-class systems began clearing 75–87% on ARC-AGI-1’s Semi-Private evaluation with substantial test-time compute, the question for ARC-AGI-1 was no longer “does this measure compositional generalization?” but “does it still isolate the gap from current frontier capability?” Once the answer was no, the benchmark needed to evolve — not to preserve a machines-can’t verdict, but because the measurement instrument had drifted out of calibration with its target.
The methodological precedent is not unusual. In wet-lab biology, assay redesign is routine when the assay’s signal-to-noise ratio degrades. The field does not call that goalpost-shifting; it calls it experimental discipline. The question for ARC-AGI is whether each successor version is genuinely targeting a new regime of generalization (calibration restored) or a moved goalpost (the score required for passage inflated without the thing being measured changing).
What the goalpost charge presses
ARC-AGI-2 was announced on March 24, 2025, about three months after o3’s ARC-AGI-1 result; ARC Prize had said in December 2024 that the new version had been in the works since 2022. ARC-AGI-3 launched on March 25, 2026, about four months after Poetiq’s refinement system, built on Gemini 3 Pro, reached 54 percent on ARC-AGI-2’s Semi-Private set. To a critic, the timing looks convenient in a cui bono sense — Chollet’s framework predicts these results, and the version cadence keeps the framework from being formally falsified. The general form of the objection is that a benchmark which re-versions in response to capability gains has lost the property of measuring something stable.
Chollet’s framework specifies in advance what would count as clearing the relevant regime — efficient acquisition of new skills on genuinely novel tasks, with action efficiency comparable to humans on first exposure. The version cadence does not change that specification. It changes the test that operationalizes it. ARC-AGI-1→2 kept the regime and recalibrated difficulty (harder tasks, same static-puzzle paradigm); ARC-AGI-2→3 is qualitatively different (interactive environments, action efficiency). The next version, ARC-AGI-4, will be the test of the discipline: a same-regime threshold inflation would confirm the goalpost charge; a qualitative regime shift would preserve the framework’s discipline.
ARC-AGI-2 retained static puzzles; ARC-AGI-3 introduced interactive environments
ARC-AGI version transitions by regime and difficulty calibration
Source: deslop.media editorial reading of ARC Prize launch posts and the goalpost critique
Note: A same-regime version with a higher threshold would support the goalpost charge. Placements are editorial readings of public statements, not measurements.
Hassabis’s different bet
DeepMind, led by Hassabis until August 2026 and since then run day to day by Koray Kavukcuoglu with Hassabis as chair, has built its capability evaluation around publicly fixed targets — competition mathematics (AlphaProof, AlphaGeometry on IMO problems), structural biology (AlphaFold on CASP). The DeepMind position is not that ARC-AGI is wrong; it is that benchmarks built around fixed targets avoid the version-cadence problem entirely. The coverage gap is real: if competition mathematics is solved, no one re-versions IMO. ARC-AGI’s response would be that competition-math evaluation does not isolate the regime Chollet’s framework cares about — compositional generalization on first contact, not deep deliberative search on a fixed problem class. Both are true; they measure different things.
What would falsify the discipline-not-evasion reading
A future ARC-AGI version that keeps the same kind of test and raises only the score threshold would be the cleanest falsification of the discipline reading. The March 2026 launch material does not show this pattern; ARC-AGI-3 is qualitatively different from ARC-AGI-2 in a way ARC-AGI-2 was not from ARC-AGI-1. In September 2026 Chollet said ARC-AGI-4 is in development for the first quarter of 2027; that version will resolve the methodological question. Until then, the framework holds — but provisionally.
Sources
- @fchollet on ARC-AGI-3 as “not a final exam” and a “moving target,” X, March 25, 2026. x.com
- “ARC-AGI-3 launch,” ARC Prize blog, March 2026. arcprize.org
- “ARC-AGI-3 Technical Report,” ARC Prize Foundation. arcprize.org
- Kyle Wiggers, “A test for AGI is closer to being solved, but it may be flawed,” TechCrunch, December 9, 2024. techcrunch.com
- Gary Marcus, “goal post shifting” post on X, via Techmeme, December 20, 2024. techmeme.com
FAQ
Did Gary Marcus accuse the ARC Prize team of goalpost shifting?
No, not in the remark usually cited. On December 20, 2024, after o3’s ARC-AGI-1 result, Marcus used “goal post shifting” for claims that o3 refuted his “wall” prediction; he wrote that the wall had not been about ARC.
How does Demis Hassabis’s evaluation philosophy differ from ARC-AGI’s?
Hassabis at Google DeepMind has favored evaluation built around publicly fixed targets — competition mathematics (AlphaProof, AlphaGeometry on IMO), structural biology (AlphaFold on CASP). These do not version on capability and avoid the moving-target accusation entirely. Chollet’s framework would answer that they measure deep deliberative search on fixed problem classes, not compositional generalization on first contact — both are real, but they measure different things.
What is Chollet’s structural defense against the moving-target charge?
The benchmark’s object is the residual gap between human and machine learning, not preservation of a frozen test. ARC-AGI-1 to ARC-AGI-2 kept the regime and recalibrated difficulty; ARC-AGI-2 to ARC-AGI-3 changed the regime, from static puzzles to interactive environments. If a future version inflates a threshold without changing the regime, the goalpost-shifting charge becomes structurally accurate. No ARC version has done so.
When was ARC-AGI-2 announced relative to o3’s ARC-AGI-1 results?
ARC-AGI-2 was publicly announced on March 24, 2025; o3-preview’s ARC-AGI-1 results (75.7% at low compute, 87.5% at high compute, both on the Semi-Private set) were announced on December 20, 2024. Critics of the version cadence read the timing as evidence of goalpost-shifting. ARC Prize had said in December 2024 that ARC-AGI-2 had been in the works since 2022, and the new version made the tasks harder rather than raising the passing threshold.