Ilya Sutskever warns pre-training is ending as AI hits 'peak data'

Since leaving OpenAI to start Safe Superintelligence, Ilya Sutskever has largely kept out of the spotlight. That changed on Friday in Vancouver, where OpenAI's former chief scientist took the stage at NeurIPS to deliver a blunt reality check to the AI community.

His message was simple: the industry has hit peak data, and pre-training as we know it is about to end.

There is only one internet

For the past few years, the prevailing AI playbook was brute-force scaling: scrape the entire open web, feed it to a massive transformer, and watch capabilities emerge.

Sutskever argued that era is over. He compared human-generated web data to fossil fuels, a finite resource that is rapidly tapping out. There is only one internet, and the industry has already consumed virtually all of it.

To explain what comes next, he turned to evolutionary biology. While most mammals follow a predictable brain-to-body mass ratio, early human ancestors broke away with a radically different scaling trajectory.

Sutskever suggested AI is at a similar inflection point. While raw pre-training scaling is hitting a wall, new architectures and training paradigms will open up a completely different path forward.

True reasoning and wild unpredictability

Instead of regurgitating patterns from endless text corpora, Sutskever expects next-generation models to become truly agentic and capable of deliberate reasoning.

Rather than guessing the next token based on what they have seen before, these systems will work through problems step by step. Crucially, he claims they will be able to understand complex concepts from very limited data without getting confused.

That leap in intelligence comes with a trade-off: unpredictability.

Sutskever warned that the more an AI system reasons, the harder it is to anticipate, drawing an analogy to advanced chess engines whose moves baffle even grandmasters. When asked about granting AI rights or freedoms down the line, Sutskever hesitated, admitting that such societal questions are deeply unpredictable, though an outcome where AI simply wants to coexist and hold rights might not be the worst ending.

Why it matters

When the godfather of the modern LLM scaling law says pre-training is dead, the whole industry has to pivot.

For tech giants, the brute-force moat built on hoarded web data and multi-billion-dollar crawl budgets is running out of juice. The battleground has already shifted toward post-training, reinforcement learning, and test-time compute, an approach evident in models like OpenAI's o1.

For developers, this is actually good news. You do not need petabytes of raw internet text to build smart workflows if future models can learn robustly from small, specialized datasets.

However, we think the real catch lies in the safety trade-off Sutskever highlighted. If reasoning makes models inherently alien and unpredictable, the black-box problem gets exponentially worse. Controlling an agent that deduces its own strategies in real time is a vastly harder challenge than steering a next-token predictor.