The real reason your LLM isn't deterministic
Thinking Machines AI just debunked the prevailing theory behind why LLM inference yields inconsistent results even with greedy sampling. While developers usually blame GPU floating-point concurrency and race conditions, modern forward passes are actually deterministic. The real culprit is a lack of batch invariance, meaning an individual request's output subtly shifts depending on which concurrent queries share the batch.