Thinking Machines AI just debunked the prevailing theory behind why LLM inference yields inconsistent results even with greedy sampling. While developers usually blame GPU floating-point concurrency and race conditions, modern forward passes are actually deterministic. The real culprit is a lack of batch invariance, meaning an individual request's output subtly shifts depending on which concurrent queries share the batch.
Z.ai just introduced its new flagship MoE models, GLM-4.5 and GLM-4.5-Air, prioritizing depth over width to enhance reasoning and autonomous agent capabilities. GLM-4.5 features 355B total parameters with only 32B active, launching with day-one support on SGLang alongside FP8 variants. Why it matters: it hits a 90.6% tool-calling success rate and outpaces Claude 4 Opus on web browsing, making it a formidable foundation for autonomous workflows.
Stanford researchers just released Tokasaurus, an open-source LLM inference engine built specifically for high-throughput batch workloads like synthetic data generation. By slashing CPU overhead, dynamically grouping shared prefixes, and optimizing pipeline parallelism for non-NVLink GPUs, it beats vLLM and SGLang by over 3x on key benchmarks.
xAI just overhauled its entire inference stack from scratch in three days using the open-source SGLang framework. Grok-2 mini is now running twice as fast with slightly improved accuracy, while the full Grok-2 model gained much smoother multi-host inference speeds.
