Stanford drops Tokasaurus, an inference engine built for raw throughput
Stanford researchers just released Tokasaurus, an open-source LLM inference engine built specifically for high-throughput batch workloads like synthetic data generation. By slashing CPU overhead, dynamically grouping shared prefixes, and optimizing pipeline parallelism for non-NVLink GPUs, it beats vLLM and SGLang by over 3x on key benchmarks.