--- title: "Moving-target discipline" publication: deslop.media byline: deslop.media editorial provenance: editorial_synthesis provenance_note: "Editorial synthesis by deslop.media editorial, assembled from the subject's public record. The subject did not sit for an interview with deslop.media and has not reviewed it." issue: 1 issue_title: The Scaling Bet grade: draft issue_version: 3 issue_version_date: 2026-10-11 piece_version: 1 first_published: 2026-10-11 reporting_current_through: 2026-10 canonical_url: https://deslop.media/feature/francois-chollet/moving-target-discipline/ cite_as: "\"Moving-target discipline,\" deslop.media editorial, Issue 1 (Draft v1, October 11, 2026)" changes: https://deslop.media/issues/1-the-scaling-bet/changes/ flag: "MCP tool flag_content, or POST https://api.deslop.media/api/flags" --- The strongest critique of ARC-AGI as a benchmark family is not technical. It is structural: each new version appears just as the field's strongest models are getting traction on the previous one. Gary Marcus's *"goal post shifting"* post of December 20, 2024 was aimed elsewhere: at readers who said o3 refuted his "wall" prediction, which he said had not been about ARC. Emily Bender, the University of Washington linguist who co-authored "Stochastic Parrots," and Melanie Mitchell, the Santa Fe Institute computer scientist, have written about how far benchmark scores can be read as evidence of general capability. Demis Hassabis's DeepMind has prioritized publicly fixed targets (AlphaProof and AlphaGeometry on IMO problems, AlphaFold on CASP) that do not version on the same cadence. The charge, if it held, would be serious — because if ARC-AGI is moving, the falsification instrument that gives Chollet's frame its empirical teeth is being preserved through evasion rather than rigor. ## What "moving target" means in Chollet's frame Chollet's defense, set out on X and in the launch material when ARC-AGI-3 opened in March 2026, is structural: the benchmark's object is not a fixed score threshold. It is the gap. ARC-AGI-1 was the first attempt at isolating a kind of generalization that scaling does not solve. When o3-class systems began clearing 75–87% on ARC-AGI-1's Semi-Private evaluation with substantial test-time compute, the question for ARC-AGI-1 was no longer *"does this measure compositional generalization?"* but *"does it still isolate the gap from current frontier capability?"* Once the answer was no, the benchmark needed to evolve — not to preserve a *machines-can't* verdict, but because the measurement instrument had drifted out of calibration with its target. The methodological precedent is not unusual. In wet-lab biology, assay redesign is routine when the assay's signal-to-noise ratio degrades. The field does not call that goalpost-shifting; it calls it experimental discipline. The question for ARC-AGI is whether each successor version is genuinely targeting a new regime of generalization (calibration restored) or a moved goalpost (the score required for *passage* inflated without the thing being measured changing). ## What the goalpost charge presses ARC-AGI-2 was announced on March 24, 2025, about three months after o3's ARC-AGI-1 result; ARC Prize had said in December 2024 that the new version had been in the works since 2022. ARC-AGI-3 launched on March 25, 2026, about four months after Poetiq's refinement system, built on Gemini 3 Pro, reached 54 percent on ARC-AGI-2's Semi-Private set. To a critic, the timing looks convenient in a *cui bono* sense — Chollet's framework predicts these results, and the version cadence keeps the framework from being formally falsified. The general form of the objection is that a benchmark which re-versions in response to capability gains has lost the property of measuring something stable. Chollet's framework specifies in advance what would count as clearing the relevant regime — efficient acquisition of new skills on genuinely novel tasks, with action efficiency comparable to humans on first exposure. The version cadence does not change that specification. It changes the test that operationalizes it. ARC-AGI-1→2 kept the regime and recalibrated difficulty (harder tasks, same static-puzzle paradigm); ARC-AGI-2→3 is qualitatively different (interactive environments, action efficiency). The next version, ARC-AGI-4, will be the test of the discipline: a same-regime threshold inflation would confirm the goalpost charge; a qualitative regime shift would preserve the framework's discipline. ## Hassabis's different bet DeepMind, led by Hassabis until August 2026 and since then run day to day by Koray Kavukcuoglu with Hassabis as chair, has built its capability evaluation around publicly fixed targets — competition mathematics (AlphaProof, AlphaGeometry on IMO problems), structural biology (AlphaFold on CASP). The DeepMind position is not that ARC-AGI is wrong; it is that benchmarks built around fixed targets avoid the version-cadence problem entirely. The coverage gap is real: if competition mathematics is solved, no one re-versions IMO. ARC-AGI's response would be that competition-math evaluation does not isolate the regime Chollet's framework cares about — compositional generalization on first contact, not deep deliberative search on a fixed problem class. Both are true; they measure different things. ## What would falsify the discipline-not-evasion reading A future ARC-AGI version that keeps the same kind of test and raises only the score threshold would be the cleanest falsification of the discipline reading. The March 2026 launch material does not show this pattern; ARC-AGI-3 is qualitatively different from ARC-AGI-2 in a way ARC-AGI-2 was not from ARC-AGI-1. In September 2026 Chollet said ARC-AGI-4 is in development for the first quarter of 2027; that version will resolve the methodological question. Until then, the framework holds — but provisionally. ## Sources 1. @fchollet on ARC-AGI-3 as "not a final exam" and a "moving target," X, March 25, 2026. [x.com](https://x.com/fchollet/status/2036879665655406944) 2. *"ARC-AGI-3 launch,"* ARC Prize blog, March 2026. [arcprize.org](https://arcprize.org/blog/arc-agi-3-launch) 3. *"ARC-AGI-3 Technical Report,"* ARC Prize Foundation. [arcprize.org](https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf) 4. Kyle Wiggers, *"A test for AGI is closer to being solved, but it may be flawed,"* TechCrunch, December 9, 2024. [techcrunch.com](https://techcrunch.com/2024/12/09/a-test-for-agi-is-closer-to-being-solved-but-it-may-be-flawed/) 5. Gary Marcus, "goal post shifting" post on X, via Techmeme, December 20, 2024. [techmeme.com](https://www.techmeme.com/241220/p16)