--- title: "The architecture that capex forgot" publication: deslop.media byline: deslop.media editorial provenance: editorial_synthesis provenance_note: "Editorial synthesis by deslop.media editorial, assembled from the subject's public record. The subject did not sit for an interview with deslop.media and has not reviewed it." issue: 1 issue_title: The Scaling Bet grade: draft issue_version: 3 issue_version_date: 2026-10-11 piece_version: 1 first_published: 2026-10-11 reporting_current_through: 2026-10 canonical_url: https://deslop.media/feature/albert-gu/architecture-that-capex-forgot/ cite_as: "\"The architecture that capex forgot,\" deslop.media editorial, Issue 1 (Draft v1, October 11, 2026)" changes: https://deslop.media/issues/1-the-scaling-bet/changes/ flag: "MCP tool flag_content, or POST https://api.deslop.media/api/flags" --- A transformer is not a fact of nature. It is a set of choices about how a sequence of tokens should be mixed, scored, and stored (the neural-network architecture Vaswani and colleagues introduced in 2017, now the basis of most frontier language models), and one of those choices is that every token a model has ever seen during a generation should sit in memory, available for any future token to attend to. That choice has a name, the "KV cache" — the running store of every previously seen token’s representation, which the transformer keeps in memory while generating. It also has a cost. Computing attention scores across a whole input scales with the square of its length during training and prefill, the property usually called "quadratic attention." During generation, a model with a KV cache does work per new token that grows linearly with the context, and the cache itself grows linearly in memory. The cost is what the capex argument of the past 18 months has, in effect, been a fight about. The choice is what almost no one in that argument has been willing to look at. A state-space model maintains, instead, a fixed-size compressed running summary of everything it has seen. It updates that summary at each step and throws the per-token receipts away. Albert Gu, the chief scientist of the audio-AI startup Cartesia, an assistant professor at Carnegie Mellon, and the researcher behind the line of architectures whose flagship is called Mamba, has spent four years arguing that this difference is not a curiosity. Since a July 2025 post on Gu’s research blog Goomba Lab made it the centerpiece, he has leaned on a single contrast: transformers are "databases" that cache every token and look it up later; state-space models are "brains" that maintain a finite-size compressed running state. Different objects, with different economics and different scaling exponents. His argument is not that transformers lose, but that quietly upstream of them sit assumptions about tokenization and attention that the next generation of frontier models will not share, and that an architecture which carries a compressed state instead of a growing cache has categorically different compute economics on language modeling once those assumptions are inspected. ## The capex and the cost curve Hyperscaler capital-spending guidance for 2026 shares a shape. In July, Amazon put its 2026 capital spending at roughly $220 billion, Alphabet at $195 billion to $205 billion, Meta at $130 billion to $145 billion, and Microsoft at roughly $175 billion, a figure it restated on a lease-accounting basis rather than by cutting planned investment. Together that is roughly $720 billion to $745 billion of companywide capital spending this year, each on the company’s own definition and not an AI-only total. None of the guidance names an architecture. The likeliest assumption behind the AI share is that today’s transformer cost curves hold. Sam Altman, OpenAI’s chief executive, has repeatedly claimed that the cost of capability falls by a factor of 10 each year, a rate he drew from the falling price per token of OpenAI’s models from GPT-4 in early 2023 to GPT-4o in mid-2024. After an agent incident in July 2026 he conditionally endorsed pacing AI development, and OpenAI backed the Pacing the Frontier petition, without retracting the cost claim. Dario Amodei, Anthropic’s chief executive, explained in February why he does not buy compute on a straight-line projection. Tenfold annual growth would take Anthropic’s revenue to $1 trillion by the end of 2027. If he bought $1 trillion a year of compute on that projection and revenue reached only $800 billion, he said, "there’s no hedge on earth that could stop me from going bankrupt." That is the same curve drawn on the revenue side. Richard Sutton is the University of Alberta computer scientist whose 2019 essay "The Bitter Lesson" (the doctrine that compute and data beat hand-engineered structure over the long run) has been read as having settled the question: stop encoding inductive biases, just scale. Gu rebutted that reading on X in August 2025: > A common takeaway from the bitter lesson is we don’t need to put effort into encoding inductive biases, we just need compute. Nothing could be further from the truth! Better inductive biases, in his telling, do not produce a constant-factor improvement that scaling will eventually swamp. They produce a different scaling exponent. The slope changes. ## Four years, two tracks S4, in October 2021, with Gu’s then-PhD-advisor Christopher Ré, the Stanford machine-learning professor and MacArthur Fellow, established that a "structured state-space" sequence model could match transformers on long-range benchmarks while running in linear time. Mamba, in December 2023, with Tri Dao, Gu’s longtime co-author and the lead on FlashAttention, added "selective state-space" — a mechanism that lets the running summary’s update rule itself depend on the input, which is what unlocked language-modeling parity. Mamba-2, in May 2024, again with Dao, introduced "state-space duality," a unifying mathematical framework that re-derives transformer attention and structured state-space models as instances of one structure, opening hardware-aware optimization across both. H-Net, in July 2025, was the live test of removing tokenization: a byte-level model that beat a compute-matched byte-pair-encoded transformer at language and got nearly four times the data efficiency on DNA. Mamba-3, in March 2026, was the consolidation paper. His current claim is hybrid — and it has not always been the version that travels on conference slides. He has never publicly claimed that pure state-space models will scale to GPT-class frontier supremacy without attention layers. His program is not hybrid-only, though. Raven, from his group in April 2026, is a linear-time recurrent model with fixed memory slots and learned, input-dependent sparse routing aimed at the recall gap. It was published alongside a variant that interleaves Raven layers with attention. The empirically converged ratio, which he says has been "independently verified by dozens of research groups," runs from three Mamba layers per attention layer to ten. The April 2025 "Gather-and-Aggregate" paper from his group studies the well-known retrieval gap of pure state-space models (they are worse than attention at exact-recall tasks like in-context lookup). It finds that hybrid models hand aggregation to their attention layers. In its layer-swap experiment, the authors took Llamba-8B, a state-space model distilled from Meta’s Llama-3.1-8B, and restored one attention layer from the original. MMLU accuracy rose from 33 to 50 percent, against 67 percent for Llama-3.1-8B, narrowing the gap rather than closing it. The architecture argument is whether the dominant sequence-mixing primitive in the next generation of frontier models stays attention-only — not whether transformers and Mamba fight to the death. Gu put the resulting position into a single sentence in the Mamba-3 launch post of March 2026: > linear layers will be predominantly used in conjunction with global self-attention layers in the future. ## Where the architecture has already landed Six independent labs have released hybrid state-space-attention models at frontier-relevant scale. The headline result sits with NVIDIA. Its Nemotron-H, released in April 2025 in 56-billion- and 47-billion-parameter forms, was trained on 20 trillion tokens in FP8; NVIDIA reports that the 47-billion-parameter variant generates 2.9 times as many output tokens per second per GPU as Qwen-2.5-72B and Llama-3.1-70B, with a 65,536-token input and 1,024 output tokens on H100s. The rest of the roster shows the strategic logic. NVIDIA is shipping it because doing so exposes new hardware-utilization curves on its own silicon. IBM, whose Granite 4.0 released in October 2025 at a nine-to-one Mamba-2-to-attention layer ratio, is the first ISO 42001-certified open large-language-model family — the inference cost-per-fixed-capability matters more in enterprise deployment than the last point of benchmark performance. Tencent’s Hunyuan-TurboS describes itself as "the first industry-deployed large-scale Mamba model"; the Technology Innovation Institute in the UAE released Falcon Mamba 7B in 2024 and has shipped the Falcon-H1 hybrid family across 2025 and 2026. Neither is constrained, the way the closed-source US frontier labs are, by sunk pretraining runs on attention-only stacks. AI21 (Jamba 1.5) and Microsoft (Phi-4-mini-flash-reasoning) round out the list. Mistral took the pure state-space route instead, with its 2024 code model Codestral Mamba. The commercial place where the architectural advantage already cashes out is voice. Cartesia ships pure-state-space text-to-speech at 90 milliseconds of model latency, by the company’s own measure. Sonic-3 launched in October 2025, alongside a $100 million round with participation from Kleiner Perkins, Index, Lightspeed, and NVIDIA, in 42 languages, with more than 50,000 customers and a Tencent Cloud partnership announced on November 25. Streaming audio is the workload where a fixed-size running state has the most to give and a key-value cache has the most to cost; that is the market that has, in 2025 and 2026, organized itself around the new architecture first. ## The strongest counter The strongest counter is the silence on the other side of the argument. The case for 2026 hyperscaler capex, roughly $720 billion to $745 billion of companywide spending, has had, this year, essentially zero direct engagement with state-space models. Keep-scaling arguments do not adjudicate the architecture question; they assume the architecture and argue about the scaling. Altman’s tenfold-a-year cost decline is built on the assumption that the cost curves the industry has been on continue to be the cost curves the industry will be on. Amodei’s $800 billion bankruptcy scenario is the same curve seen from the revenue side. The empirical version of the counter is harder to dismiss. Hybrid state-space-attention has been available at production scale for more than two years. The closed-source US frontier flagships — the GPT-5 family, Claude Opus 4.x, Gemini 3.x — had not publicly confirmed any state-space or Mamba component, and neither had Meta’s open-weight Llama 4. OpenAI’s GPT-6 releases that followed (Astra in early September 2026, Sol and Luna on September 22, GPT-6.1 Sol on September 29) disclosed none in their launch materials. Nor did the newer flagships from the other three: Meta’s Muse Spark 1.3 on September 2, Anthropic’s Claude Opus 5.5 on September 22, and Google’s Gemini 4 Argon, announced on September 30 for a restricted first group of users. Either the costliest place in the industry is the slowest place in the industry, which is Gu’s implicit reply, or state-space proponents have overestimated the architecture’s edge. One explanation is switching cost. Frontier teams have stacks calibrated to attention. Switching the sequence-mixing primitive forces a rebuild of distributed-training kernels, KV-cache infrastructure, reinforcement-learning rollout machinery, and the inference serving stack: months to quarters of work that does not directly improve a benchmark. Pretraining raises the price. Anthropic, OpenAI, and Google run pretraining for months on hundreds of thousands of GPUs, so an architectural change that risks any of that run costs more there than at NVIDIA’s research-and-product pace. NVIDIA, IBM, AI21, Tencent, Microsoft and the Technology Innovation Institute have stronger incentive to expose new hardware-utilization curves than the frontier flagships do, because the marginal benchmark point is not the binding constraint on their go-to-market. The other is infrastructure substitution. KV-cache compression, multi-query and grouped-query attention, mixture-of-depths routing, speculative decoding, FlashAttention-3, and FP4 quantization on Blackwell silicon are, between them, a stack of tricks that compound. If they compound enough, attention’s quadratic cost stops binding at the workloads frontier flagships actually serve. Epoch AI’s September 2026 analysis puts the cost of a given benchmark score falling 47 percent a quarter overall, and 66 percent a quarter for scores that are newly state of the art; that pace makes the architectural-replacement case harder, not easier, the longer it goes on. Neither explanation says the architecture does not matter. Both say it has not yet mattered enough to overcome the switching cost at the costliest switching point. The next year will test whether it will. ## The shape of the disagreement Gu sits between two more famous critics on the architecture question. Yann LeCun, Meta’s former chief AI scientist, who left at the end of 2025 to found AMI Labs, argues for substrate-level redesign: world models and joint-embedding predictive architectures that learn to predict future states of the world rather than the next token. The disagreement is with what is being modeled — token streams versus a learned latent of the world. Ilya Sutskever, who co-founded OpenAI and now runs Safe Superintelligence, has called publicly, through 2025, for a return to research over scaling, and in July 2026 SSI and NVIDIA announced a partnership to increase SSI’s compute by an order of magnitude. The disagreement, in his framing, is recipe-level: which objectives, which data, which post-training regime. Gu’s disagreement is narrower than either. It is sequence-mixing-level. The thing he wants to swap out is the primitive that decides how a model combines information across positions in a sequence; the thing he wants to leave alone is most of the rest of the stack. Because Gu’s claim is narrower, it is testable. In early 2024, Samy Jelassi and the Harvard team behind "Repeat After Me" showed that pure state-space models fail at exact-copy retrieval at lengths where transformers do not. Gu’s "Gather-and-Aggregate" paper, in April 2025, traced a related retrieval weakness to a few aggregation heads and showed that restoring one attention layer in a distilled state-space model recovers part of the lost accuracy. The "Bitter Lesson" reading he is rebutting is the one that says inductive biases are a category mistake; he is arguing the opposite, that the right inductive biases set the slope of the scaling curve and that "FLOP-for-FLOP," "pound-for-pound," a hybrid is on a better one. ## What the next year will test The test is disclosure. None of the September 2026 launch announcements listed above disclosed a state-space layer, so the test passes to the releases that follow. If a frontier flagship from OpenAI, Anthropic, Google, or Meta ships in the next 12 to 18 months with a state-space layer in the stack — at any ratio, in any role — the keep-scaling camp’s transformer-only assumption will have been falsified at the costliest switching point in the industry. That alone would not show that 2026 capex was misjudged. Whether the AI share was planned for the wrong architecture would turn on what the new designs cost to serve, per token, against the attention-only stacks they replace. If no such flagship does, and attention-only stacks with infrastructure-substitution tricks keep cutting the cost of frontier-grade generation at the pace Epoch measured in September 2026, the architecture-matters case will have to argue itself on a longer time horizon than this capex cycle. The July 2025 Goomba Lab post that introduced the "brains" and "databases" contrast was titled "On the Tradeoffs of SSMs and Transformers," and the central organizing idea — that architectures encode different bets about what is worth caching and what is worth compressing, and that the right bet depends on the workload — is itself a hedge. A "brain" is the right object for streaming audio and very long context, where the cost of retaining everything outweighs the value. A "database" fits in-context retrieval, where exact recall over a recent window is the workload. Hybrids exist because most real workloads are mixtures. The keep-scaling camp has been arguing about how much compute to throw at a settled architecture; the question Gu has spent four years posing is whether the architecture is settled. Six labs say it is not. The September launch announcements from the four US frontier labs did not address it. That July 2025 post put it this way: > I think Transformers are amazing and attention is truly a fundamental modeling primitive. But I also think it’s clear that they, by themselves, are not the final solution.