Sutton against Chollet, on different axes

The sharpest argument among the skeptics of LLM scaling is between two of them who agree that current pretrained transformers do not reach AGI, and disagree on almost everything else.

Richard Sutton and François Chollet, the Keras creator and ARC-AGI benchmark designer who co-founded Ndea with Mike Knoop of Zapier, have no direct exchange in the record this piece draws on. The disagreement has to be reconstructed from what each has published. Both reject the claim that scaling current pretrained LLMs delivers artificial general intelligence. Each locates the failure somewhere different.

Chollet’s published position runs through the ARC benchmark. “On the Measure of Intelligence,” his 2019 paper, distinguishes memorized skills, which are static and task-specific, from fluid general intelligence — the ability to understand something you’ve never seen before on the fly. The ARC Prize launched in 2024; the technical report (arXiv 2412.04604) documented the field’s struggle to clear it through pretraining alone. ARC-AGI-2 launched March 24, 2025 alongside the ARC Prize 2025 launch, with the original ARC-AGI-1 reduced to a calibration baseline. ARC-AGI-3 launched at Y Combinator on March 25, 2026.

Chollet’s most quoted technical claim arrived at AI Startup School on June 16, 2025. “Even after a 50,000x scale-up of pre-trained base [models],” he said, “the performance in ARC stayed near zero” — the empirical observation that frames the rest of his argument. Chollet then drew the inference: fluid intelligence does not, in his reading, emerge from scaling up pre-training. His current research vehicle, Ndea, is building “a lifelong-learning, self-improving, DL-guided program synthesis engine.”

For Chollet, the missing piece is compositional generalization — the architectural capacity to assemble a working program for a novel task on the fly. The path runs through program synthesis, neural-guided search, and architectural priors that build flexibility in. His Y Combinator quote on AGI: a system that “can approach any new problem, any new task, any new domain and make sense of it… with the same degree of efficiency as a human could.”

For Sutton, the missing piece is continual experience — an agent acting in an environment, predicting consequences, being surprised, updating. The path runs through scaled experiential reinforcement learning, the “OaK architecture” — “Options and Knowledge,” a model-based RL agent with continual learning and an open-ended abstraction cycle — and a continual-learning algorithm that does not yet fully exist. His own framing: “Squirrels don’t go to school.” Embodied animals learn from streamed sensory input and reward; current LLMs do not.

What each one would say about ARC

Chollet measured frontier models on ARC-AGI-2 and ARC-AGI-3 and, at ARC-AGI-3’s March 2026 launch, reported them near zero on the harder benchmark. His read then: scaling pretraining does not produce fluid intelligence, and ARC’s resistance was the proof. That resistance weakened in September 2026, when ARC Prize reported GPT-6 Astra at 62.7% on ARC-AGI-3’s Semi-Private set under the Standard harness at max reasoning effort. Chollet responded that his views had shifted after both o3 and Astra. He described o3 as a qualitative leap to “solid fluid intelligence” and called Astra a “step-function change” and a “major breakthrough,” without abandoning his view that scaling base models alone does not produce fluid intelligence. The ARC of Progress survey (arXiv 2603.13372, March 2026) documents “performance degradation across versions is consistent across all paradigms: program synthesis, neuro-symbolic, and neural approaches all exhibit 2-3x drops from ARC-AGI-1 to ARC-AGI-2.”

Sutton’s structural response, inferable from his 2025 corpus, is that the models being measured are imitation systems trained on human-generated data. They are exactly the systems his “Bitter Lesson” reading says will be superseded. ARC’s resistance to those systems is consistent with his thesis. The question is not whether pretrained transformers clear ARC. The question is whether an agent learning continually from experience would.

The split between the two arguments is test-time training. The ARC Prize 2024 Technical Report documented that the first-place ARChitects team scored 53.5% on the private ARC-AGI-1 evaluation set using test-time training — a runtime adaptation method that updates model parameters on the test instance itself, against a small handful of demonstration pairs. Test-time training is now the dominant winning strategy on ARC-AGI-1.

By April 2026, Confluence Labs reported 97.92% on ARC-AGI-2’s public evaluation set at $11.77 per task (per its GitHub repository, corroborated by Imbue’s February 27, 2026 write-up) — a multi-agent program-synthesis harness result, not comparable to base-model or verified semi-private scores, and again via specialized scaffolding rather than base model scaling.

Sutton’s runtime-learning emphasis finds partial, accidental confirmation in those results, even though the underlying base models are LLMs. The ARChitects result clears ARC-AGI-1 by spending more compute at runtime, on the specific task at hand, learning from the demonstration pairs the way Sutton would say a real intelligence learns from environmental feedback. It does not clear the benchmark by scaling pretraining further.

Chollet has acknowledged the result without conceding the architectural point. His own framework treats test-time adaptation as one tool inside a larger program-synthesis story. Sutton has not commented publicly on ARC by name. Both sides can read the same result as confirmation. The frontier scores below 1% at the March 2026 launch of ARC-AGI-3 — a benchmark that requires interactive agentic intelligence rather than offline classification — moved the goalpost back to the place where Chollet says architectural priors will keep mattering and Sutton says continual experience will keep mattering.

Where the disagreement lives

Generality, for Chollet, is downstream of architectural priors that bias the system toward compositional reasoning. Generality, for Sutton, is downstream of continual experience and the abstractions an agent builds for itself in the course of acting. Chollet wants more structure built in. Sutton wants more experience.

This is the deeper layer of the “Bitter Lesson” dispute. Chollet’s program sits close to the symbolic-AI tradition Sutton’s 2019 essay was written against — “we want AI agents that can discover like we can, not which contain what we have discovered.” Chollet would reply that compositional priors are not contents-of-discovery; they are the substrate that lets discovery happen at all. Sutton’s “Bitter Lesson” line about “the actual contents of minds are tremendously, irredeemably complex” cuts the other way: even elegant priors like symmetry or compositionality, in his frame, are the kind of design-time injection that the long arc of AI research has shown to lose to scaled learning.

Dario Amodei, Sam Altman, and the major-lab consensus sit at a third position: scale pretraining and inference-time compute on the current paradigm and let generality emerge. Chollet’s ARC-AGI-3 launch numbers were the sharpest published evidence against that position, until GPT-6 Astra’s September 2026 results cut the other way. Sutton’s “Era of Experience” frame is the most direct published critique of why pretraining-as-substrate is the wrong unit even when it works. The two bears differ on which unit to replace it with.

The framing question Patel asked

In his September 2025 interview, Dwarkesh Patel pushed Sutton on the squirrel argument. Sutton’s position was that a squirrel’s intelligence — flexible, embodied, continually learning — is closer to general intelligence than any current LLM. “Supervised learning is not something that happens in nature.” Cumulative-culture research, Patel pushed back, suggests human cognition is downstream of culture; humans without culture are not particularly impressive squirrels.

Sutton’s reply was that culture is the loop, not the substrate. The substrate is on-the-job learning and prediction-of-consequences. Culture rides on top of that loop. A system that learns only from imitation has no loop; it has only the trace. The same move is available against a pretrained LLM: the model has the trace of human cognition embedded in its weights, but no loop of its own.

Gary Marcus, the NYU emeritus professor of psychology, has held a third bear position since 2022 — “Deep Learning Is Hitting a Wall” — and has read inference-time-compute results as a quiet pivot toward neurosymbolic AI rather than as scaling vindication. Marcus’s bear case is closer to Chollet’s; both want more architecture, less reliance on raw learning. Marcus has staked a 10:1-odds wager with Miles Brundage on the AGI question for end of 2027 (December 30, 2024), tracked on Metaculus question 31246: if AI systems clear 8 of 10 tasks, Marcus donates $2,000; if they fall short, Brundage donates $20,000. Sutton is doctrinally further from Marcus than from Amodei on the question of how much architecture matters at all. Sutton’s “Bitter Lesson” reading and Marcus’s neurosymbolic frame are not adjacent positions.

What would settle it

Two benchmarks have carried most of the weight. ARC-AGI-3 measures novel-task adaptation in interactive environments; the ARC Prize 2026 framework runs its prize competitions on Kaggle and requires winning solutions to be open-sourced, while its official leaderboard separately evaluates commercial API models. METR’s time horizon is the length of software task, timed by how long human experts take, that a model completes with 50% probability. METR Time Horizon 1.1, released January 29, 2026, estimated a doubling time of 130.8 days for models from 2023 onward, against a 7-month long-run trend. On September 8, 2026, METR noted that the public series was no longer actively updated.

Sutton expects a continual-learning agent in the “OaK” frame, given enough compute and environmental contact, to eventually clear both. Chollet expects scaling-only agents to keep clearing benchmarks and failing the next one he builds, while Amodei, who told Dwarkesh Patel in February 2026 that his model of scaling had not changed since 2017, expects further scaling to clear them by brute force. The two bears agree that scaling pretraining alone falls short. They disagree on what should replace it. Later results on ARC-AGI-3, and on whatever benchmark Chollet builds next, will reward at most one of the two readings, and could reward neither.

Sources

• Chollet, AI Startup School, June 16, 2025
• ARC Prize 2024 Technical Report (ARChitects 53.5% TTT result)
• ARC Prize Foundation, ARC-AGI-3 technical report, March 2026
• Sutton on Dwarkesh Patel, September 26, 2025
• Dario Amodei, Dwarkesh Patel interview, February 13, 2026
• Silver and Sutton, “Era of Experience,” April 2025
• METR Time Horizon 1.1, January 29, 2026
• Marcus and Brundage 10-task wager, December 30, 2024