The real reason your LLM isn't deterministic

Ask ChatGPT the exact same prompt twice with the temperature set to zero, and you will still get different answers. For years, the AI world shrugged and blamed it on modern hardware: thousands of GPU threads racing each other, combined with floating-point math where order changes the outcome.

Now, Mira Murati's stealthy startup, Thinking Machines Lab, is calling BS on that explanation.

Researcher Horace He published the lab's first research post, "Defeating Nondeterminism in LLM Inference." The post dismantles the widespread "concurrency plus floating point" hypothesis and reveals why today's models act so randomly.

Debunking the concurrency myth

He started with a simple experiment: run the exact same bfloat16 matrix multiplication on a GPU one thousand times, and the results are bitwise identical every single run. If GPU parallelism and floating-point errors inherently cause randomness, that shouldn't happen.

Floating-point math indeed breaks associativity, meaning numbers added in a different order produce slightly different results. But He points out that standard LLM forward passes almost never use non-deterministic operations like atomic adds. Modern inference libraries already avoid cross-thread race conditions by parallelizing across the batch dimension instead.

So why do inference servers like vLLM or commercial APIs produce varying tokens? He argues the culprit is not the kernels themselves, but how those GPU kernels are orchestrated and stitched together during serving. By tightening control over this execution pipeline, the team argues determinism is entirely achievable.

The real goal: smoother RL

While reproducible outputs are crucial for science and enterprise compliance, Thinking Machines Lab has an even bigger prize in mind: reinforcement learning.

When training models via RL, unpredictable outputs inject noise into the reward signals. Making generations perfectly consistent gives models cleaner feedback, smoothing out post-training runs.

Murati, OpenAI's former CTO, raised $2 billion at a $12 billion valuation for the startup, which reportedly plans to use RL to build custom corporate models. Murati previously teased a first product arriving in the coming months for researchers and startups, and the post marks the kickoff of a public technical series dubbed Connectionism.

Why it matters

Deterministic inference sounds like a niche systems problem, but it is one of the biggest bottlenecks holding back mission-critical AI. If you are building automated legal compliance, automated theorem provers, or deterministic code-refactoring agents, zero-temperature randomness is an absolute nightmare.

We think tackling this layer first is a remarkably sharp move. Instead of simply building another massive foundation model, Thinking Machines Lab is fixing the foundational plumbing. If they can eliminate noise during RL, their post-training efficiency could leap ahead of competitors burning millions on noisy runs.

The real test will be latency. The industry relies heavily on tricks like dynamic batching and speculative decoding to keep GPU serving fast, both of which introduce execution churn. Proving determinism works in a static script is one thing; delivering it at scale without cratering throughput will be the real hurdle Murati's team has to clear.