The AI scaling wall is getting real

For the past two years, the AI playbook seemed simple: throw more computing power and internet data at a model, and watch it get exponentially smarter.

That free ride might be coming to an end.

OpenAI, Google, and Anthropic are all running into significant hurdles trying to build their next generation of frontier models, according to reports from Bloomberg and other outlets. The core problem is as straightforward as it is alarming: the industry is running out of fresh, high-quality data to train on.

Orion falls short of the hype

Take OpenAI's upcoming flagship model, code-named Orion. The company finished an initial round of training back in September, hoping for a leap that would dwarf current offerings and push closer to human-level intelligence.

Instead, early internal tests show Orion fell short of targets. People familiar with the matter noted that as of late summer, the model stumbled on coding tasks that fell outside its training data. Insiders reportedly feel Orion's performance gains over GPT-4 are nowhere near the dramatic leap seen from GPT-3.5 to GPT-4.

OpenAI is not alone in this slowdown. Google's latest Gemini iterations have missed internal performance goals, while Anthropic has held back the release of its anticipated Claude 3.5 Opus, hinting at similar wall-hitting across the board.

Running out of web data and cash

Large language models have effectively hoovered up the public internet. What remains is locked behind paywalls, guarded by publishers, or tied up in court. OpenAI CEO Sam Altman has admitted that training frontier models without copyrighted material is practically impossible, even as the company fights off lawsuits from content creators.

At the same time, the economics are becoming brutal. Training and maintaining advanced systems is burning through capital at an unprecedented rate. OpenAI projected $5 billion in losses before raising $6.6 billion from Microsoft, Nvidia, and others at a $157 billion valuation. Analysts do not project profitability until 2029, and OpenAI faces pressure to convert into a for-profit entity or risk returning investor cash.

Altman maintains that artificial general intelligence is still achievable with today's technology, but acknowledging that reaching it will require billions in infrastructure feels very different when the models themselves are showing diminishing returns.

Why it matters

For developers and everyday users, the takeaway is clear: do not wait around for a magical next-gen model to solve all your problems overnight. The days of effortless generational leaps every six months are likely over. Value will shift from waiting on foundation models to building better scaffolding, memory, and specialized agent workflows around the models we already have.

For the tech giants, the brute-force era of the Scaling Law is hitting an empirical wall. Pouring more text into a bigger transformer is no longer yielding automatic breakthroughs. We think this forces the industry into a much more difficult phase of research, where teams must rely on synthetic data, test-time reasoning, and post-training tricks rather than just crawling the open web.

The real danger is financial. Silicon Valley has priced these companies as if radical intelligence jumps were guaranteed every year. If massive clusters of GPUs produce only incremental improvements, the staggering burn rates and delayed profitability will eventually test the patience of even the deepest-pocketed backers.