The Shape of Models to Come

There is a line in the config of almost every product I run that names a model. Two or three times a year I change that line, run my test prompts, and watch the behavior of my whole product shift underneath me. It is the strangest dependency I have ever built on: a component I cannot inspect, produced by a process I only half understand, replaced on someone else's schedule.

So I have a practical reason to care about model architecture. Not academic curiosity — supply-chain anxiety. When I read about what comes after the transformer, I am really asking one question: what am I actually betting on?

The transformer is a format, not a law

I come from broadcast, an industry that lives through format transitions. SD to HD. SDI to IP. Every transition looked impossible until it was suddenly finished. The lesson broadcast taught me is that dominant formats are not laws of nature; they are equilibria of cost, tooling, and inertia. They hold until the economics flip.

The transformer feels like that to me. Attention — every token comparing itself to every other token — is beautifully general and famously expensive, with cost growing roughly with the square of sequence length. For years that cost was worth paying because nothing else matched the quality. That is a market condition, not a theorem.

Four contenders, honestly assessed

From my amateur seat, four shapes seem worth watching.

Mixture-of-experts is the least speculative because it has already won. Instead of pushing every token through the full network, a router activates a few specialized sub-networks per token, so you get a huge model's capability at a fraction of the compute per token. Most frontier labs stay quiet about internals, but the open-weight models tell the story: sparsity is now normal.

State-space models like Mamba attack the quadratic cost directly, carrying a compressed running state through the sequence so cost grows linearly with length. Pure versions have real weaknesses — precise recall is harder when you cannot look back at everything — which is why the practical outcome so far is hybrids: mostly state-space layers, with a few attention layers kept where they earn their cost. That pattern, keeping the expensive thing only where it pays, feels durable to me.

Reasoning-native training changes what we optimize for rather than the wiring: models rewarded for long chains of thought that reach correct answers, spending variable compute at inference time. This is the one I feel daily as a builder, because it broke my cost model. I used to pay per token of answer; now I pay per token of thinking, and latency became a product decision.

World models are the speculative one: systems that learn to predict how an environment evolves, not just which word comes next. If agents are going to act in software and the physical world, something like an internal simulator seems necessary. I hold this one loosely. It might be the future; it might be a decade of demos.

Bet on interfaces, not internals

Here is the other thing broadcast taught me. Cameras went from tubes to chips to computational pipelines, and almost nobody's control room collapsed, because the grammar of production — preview, program, cut — stayed put. The durable layer was the interface, not the internals.

I think the same holds here. I never touch an attention matrix; I touch an API that accepts text and tools and returns text and tool calls. Every architecture above competes underneath that same surface. Whichever wins, my code mostly does not care. What changes is the shape of the cost curve, how far into a long context I can trust recall, and how much latency my UX has to absorb.

So my practical bets look like this. Design for variable latency, because reasoning-style inference makes response time a distribution, not a number. Treat context as cheap but attention within it as unreliable, and keep my own retrieval and state rather than dumping everything into the window. And keep the model swappable behind one thin seam, because the one certainty is that the line in that config file will change again.

What I would actually predict

If I had to guess: no clean successor. The transformer will do what most dominant formats do — dissolve into hybrids rather than lose a war. Sparse, partly state-space, trained to reason, perhaps wrapped around something that earns the name world model. Asking what replaces the transformer may be like asking what replaced the electric motor. Nothing did. It disappeared into everything.

The architecture will keep changing shape. The discipline of building well on top of it does not have to.