Thinking Machines AI just debunked the prevailing theory behind why LLM inference yields inconsistent results even with greedy sampling. While developers usually blame GPU floating-point concurrency and race conditions, modern forward passes are actually deterministic. The real culprit is a lack of batch invariance, meaning an individual request's output subtly shifts depending on which concurrent queries share the batch.
Stanford researchers just released Tokasaurus, an open-source LLM inference engine built specifically for high-throughput batch workloads like synthetic data generation. By slashing CPU overhead, dynamically grouping shared prefixes, and optimizing pipeline parallelism for non-NVLink GPUs, it beats vLLM and SGLang by over 3x on key benchmarks.
Google just detailed how to deploy Gemma models on Dataflow using vLLM for streaming inference. By leveraging continuous batching and Dataflow's model manager, developers can spin up an end-to-end serving pipeline in minimal code, boosting inference throughput by over 23x in benchmarks.
Red Hat just signed a definitive agreement to acquire Neural Magic, an MIT spinout known for accelerating gen AI inference and leading the open-source vLLM project. The move aims to make cross-platform LLM serving vastly cheaper and easier across enterprise hybrid clouds.
