---
title: "The \"jagged\" critique survives the leaderboard; the quantitative claim does not"
publication: deslop.media
byline: deslop.media editorial
provenance: editorial_synthesis
provenance_note: "Editorial synthesis by deslop.media editorial, assembled from the subject's public record. The subject did not sit for an interview with deslop.media and has not reviewed it."
issue: 1
issue_title: The Scaling Bet
grade: draft
issue_version: 3
issue_version_date: 2026-10-11
piece_version: 1
first_published: 2026-10-11
reporting_current_through: 2026-10
canonical_url: https://deslop.media/feature/ilya-sutskever/jaggedness-and-the-generalization-claim/
cite_as: "\"The \"jagged\" critique survives the leaderboard; the quantitative claim does not,\" deslop.media editorial, Issue 1 (Draft v1, October 11, 2026)"
changes: https://deslop.media/issues/1-the-scaling-bet/changes/
flag: "MCP tool flag_content, or POST https://api.deslop.media/api/flags"
---

Sutskever's argument against contemporary models, in his November 2025 interview with the podcaster Dwarkesh Patel, does not arrive as a technical claim. It arrives as a complaint.

Ilya Sutskever, who co-founded OpenAI and now runs Safe Superintelligence, was describing what he takes to be the deepest tell about where the field actually is when he gave Patel the example:

> You go to some place and then you get a bug. Then you tell the model, "Can you please fix the bug?" And the model says, "Oh my God, you're so right. I have a bug. Let me go fix that." And it introduces a second bug.

Tell the model about the second bug, he goes on, and the model fixes it and brings back the first.

The argument isn't "models make mistakes." Models making mistakes is uninteresting. The argument is that the failure has a shape — apologetic, eager, locally competent, globally lost — and that the shape is what one would expect if the system had memorized the surface of debugging without learning whatever humans are doing when they keep two bugs in their head at once. Sutskever's gloss on this, also from the Dwarkesh interview, is that today's models "generalize dramatically worse than people. It's super obvious. That seems like a very fundamental thing."

That sentence carries the weight of his "age of research" periodization. Without it, the periodization is a story about taste and aesthetic dissatisfaction with the recipe. With it, the periodization has a technical floor: the recipe is exhausted not because the field ran out of GPUs but because the thing the recipe produces fails at the kind of transfer humans do effortlessly, and no amount of additional pretraining data will fix that, because the gap is in the objective, not the data.

The 2023–2026 leaderboard record is harder on the claim than the prose is.

## The qualitative argument is sharp

The bug-loop is one piece. The second is the AlphaZero analogy from his NeurIPS 2024 "Test of Time" talk: as systems reason more, they become more unpredictable, and AlphaZero's late-game choices already exceed human chess intuition in ways that feel less like better-chess-player and more like alien-chess-player. The third is the two-students analogy.

Two students, Sutskever says. One has put in ten thousand hours on competitive programming, drilling endlessly, and is excellent. The other has put in a hundred hours and is "with it" — picks things up, has taste, can transfer. The phrase "with it" matters. Sutskever uses it to mark a category that brute optimization does not produce. The grinder will beat the with-it kid on the next contest. Over a career, the with-it kid does the new thing.

These three together give a coherent qualitative picture. Models are grinders. Their gains on benchmarks are real, but they reflect optimization against benchmark surfaces, not the kind of compositional understanding that lets a hundred-hour student carry a new domain without scaffolding. The bug-loop is what shows up when a grinder hits a problem that requires holding two states in tension. The AlphaZero unpredictability is what shows up when grinder optimization runs past the regime humans grind in. The taste-versus-grind frame ties the other two together.

The qualitative case is hard to argue with on its own terms. The behavior is real; engineers who have shipped agentic deployments at frontier labs and their largest enterprise customers have lived the bug-loop.

## The benchmark record is harder on the claim

The 2023–2026 benchmark trajectory cuts against the claim that the gap is widening. GPQA Diamond moved from 39 percent in late 2023 past the PhD-expert baseline into the 90s by 2026: Gemini 3 Deep Think scored 93.8 percent, and Claude Mythos Preview, published by Anthropic on April 7, 2026, scored 94.6 percent. SWE-bench Verified climbed from roughly 50 percent in early 2025 to 93.9 percent for Claude Mythos Preview in that April release. On the harder SWE-bench Pro, Mythos Preview scored 77.8 percent, and in July 2026 an OpenAI audit found that about 30 percent of Pro's 731 public tasks had design or grading defects. Gemini 3 Deep Think reached a Codeforces Elo of 3455, past elite-human, and IMO 2025 performance reached the gold-medal standard at 35 of 42, certified by IMO President Gregor Dolinar, end-to-end in natural language inside the 4.5-hour competition limit.

The harder external tests are more equivocal. ARC-AGI-3 opened in March 2026 with frontier models below 1 percent against a 100-percent human baseline; on September 2, 2026, ARC Prize reported GPT-6 Astra at 62.7 percent on its Semi-Private set under the Standard harness at max reasoning effort. METR's Time Horizon 1.1 reports a post-2023 doubling time of 130.8 days for the 50-percent reliability time-horizon, with Claude Opus 4.6 reaching 14 hours 30 minutes by February 21, 2026.

<!-- figure:primary -->

## Two readings of one record

A reader who only has Sutskever's vocabulary will look at GPQA at 94 and see the saturation as confirmation of his point: contamination, eval-aligned RL, optimization against the surface of the benchmark rather than the substrate it was trying to test. A reader who only has the leaderboard will look at IMO gold and METR's 4.3-month doubling and see his claim as a denial of evidence.

The qualitative claim — that models exhibit a recognizable failure mode under sustained agentic load that humans do not — survives the leaderboard. The bug-loop is real, OpenAI's own audit found that every frontier model could reproduce verbatim gold patches for some SWE-bench Verified tasks (so part of the saturation reflects contamination), and ARC-AGI-3 was a sharp external test of compositional generalization that frontier models failed at launch in March 2026, at under 1 percent, much as they had failed ARC-AGI-2 a year earlier; on September 2, 2026, ARC Prize reported GPT-6 Astra at 62.7 percent on its Semi-Private set under the Standard harness at max reasoning effort. The ARC of Progress survey out of February 2026 found a consistent two- to threefold drop from ARC-AGI-1 to ARC-AGI-2 across all paradigms — neural, neurosymbolic, program-synthesis. Something architectural is going on.

The strongest specific empirical anchor for the qualitative diagnosis is not in Sutskever's own writing. It is in Anthropic's. The Gema, Hägele, Chen et al. paper on inverse scaling in test-time compute (arXiv:2507.14417, July 2025), published by researchers at Anthropic — a lab whose business rests on scaling — documents five named failure modes that surface inside the test-time-compute regime as the reasoning budget grows: distractor sensitivity, spurious-correlation drift, deductive-focus collapse, problem-framing overfit in OpenAI's o-series, and concerning self-preservation expressions in Claude Sonnet 4 under extended reasoning. The paper does not claim the test-time-compute curve fails in aggregate — it argues the curve survives in the average — but it documents five distinct regimes in which longer thinking makes the model worse, not better. Those five failure modes are the technical-floor reading of Sutskever's jaggedness diagnosis written in benchmark-evaluable form. The paper is published by the lab whose reasoning models the diagnosis is most directly aimed at, and it concedes the recognizable failure shapes. The bug-loop diagnosis has a peer-reviewed analogue inside the bull camp.

The quantitative claim — that the gap is widening, or large in absolute terms, or that no amount of additional reasoning compute helps — does not survive the leaderboard. GPQA at 94 is real, even allowing for contamination. IMO gold inside the time limit is real. METR's doubling is real, even allowing for the small-sample caveat at the top of its distribution (only 14 tasks in the 1-to-4-hour bin). The inverse-scaling paper, again, is the strongest available evidence pointing the other way — and even it concludes that the test-time-compute curve is real in aggregate. The qualitative reading has a peer-reviewed empirical foundation; the bull-camp curve has a peer-reviewed empirical foundation; the two are not arguing about the existence of either. What follows depends on a question Sutskever has not answered.

## The metric problem

If models generalize dramatically worse than people and the gap is fundamental, what observation in 2026 would falsify the claim? A specific score on a specific benchmark? A specific time-horizon? A specific failure-mode rate at a specific reliability threshold? Sutskever has not said. Across his NeurIPS 2024 talk, his SSI founding statement, his July 2025 leadership-change post, and the November 2025 Dwarkesh interview, he offers aesthetic criteria for what good research looks like ("beauty, simplicity, elegance, correct inspiration from the brain"), structural arguments about what kind of company can do it (no products, "scale in peace"), and concrete failure descriptions like the bug-loop. He has not named a benchmark whose saturation he would accept as showing he was wrong. He has not named a time-horizon whose crossing he would treat as decisive. He has not named a failure-mode whose disappearance he would count as the qualitative gap closing.

The charitable reading of that absence is that it is principled. Sutskever's point about RL is that practitioners scale RL on the same eval-aligned environments that produce the bug-loop he describes. If the existing benchmark surface is itself the artifact of the recipe he calls exhausted, naming a benchmark would amount to capitulating to the framework he is rejecting. ARC-AGI-3, with its interactive design, was the strongest external version of that reading: its design rests on the premise that what one wants to test is not in the benchmark surface as currently constructed. By September 2026, though, an API-served frontier model, GPT-6 Astra, had reached 62.7 percent on its Semi-Private set under ARC Prize's Standard harness.

That charitable reading has weight. SSI ships no product, and its own site listed no paper or technical report as of October 10, 2026. That posture is consistent with a lab that treats the existing benchmark surface as the wrong test. If SSI were running its models against GPQA and SWE-bench, it would presumably publish.

But the principled refusal still leaves the field without a falsifier from the researcher who has placed the largest direct bet on the recipe being over. Noam Brown, OpenAI's vice president of research, whose work powered the o-series move into inference-time compute, posted on April 23, 2026: "A hill that I will die on: with today's AI models, intelligence is a function of inference compute. Comparing models by a single number hasn't made sense since 2024. What matters is intelligence per token or per $." That reframing is the strongest bull response to the jaggedness claim, and it has a method attached. Brown's frame says: take any task, plot intelligence-per-dollar against the dollar axis, watch the curve over time. If the curve flattens or inverts, there is evidence; if it keeps marching, there is not. The frame is testable in a way Sutskever's is not.

Dario Amodei's seven-factor model from his February 2026 Dwarkesh interview ("my top level model of scaling is the same as it was in 2017") is similarly measurable: compute, data, data quality and distribution, length of training, an objective function that scales, and two normalization-and-conditioning factors. Each of the seven can be tracked. Anthropic's own inverse-scaling paper engages directly with the regime where Sutskever says the recipe fails, and finds the curve survives in aggregate. The bull voices are showing their work in a way the central claim under Sutskever's periodization is not.

Nearly two years after the NeurIPS talk and almost a year after the Dwarkesh interview, the qualitative argument — bug-loops, AlphaZero unpredictability, the with-it student — is sharp and recognizable. The quantitative argument is contested by a leaderboard that has moved further than the bear case predicted on most measures and further than the bull case predicted on a few. His qualitative diagnosis survives the leaderboard. The metric that would settle the quantitative claim has not been published.