Stanford drops Tokasaurus, an inference engine built for raw throughput

Tokasaurus is an open-source LLM inference engine from Stanford built from the ground up for one specific goal: squeezing maximum throughput out of heavy batch workloads.

What it actually does

When you are generating synthetic data, running offline evals, or sampling thousands of reasoning paths for reinforcement learning, individual request latency barely matters. What counts is how cheaply and quickly the engine finishes the entire batch.

Tokasaurus focuses entirely on total token throughput. On standard benchmarks like ShareGPT and high-prefix-sharing tests like Large Language Monkeys, it consistently outperforms vLLM and SGLang, delivering more than double the throughput on multi-sample math tasks.

When scaling up across multiple GPUs, it really flexes its muscles on budget hardware. Benchmarked with Llama-3.1-70B on eight L40S GPUs without fast NVLink interconnects, Tokasaurus beats the pipeline parallel implementations of both vLLM and SGLang by over 3x.

Under the hood and specs

The engine hits these numbers using three architectural optimizations.

First, it eliminates CPU bottlenecks with an adaptive manager. In small model inference, the CPU often struggles to feed the GPU fast enough. Tokasaurus constantly monitors queue depth, automatically skipping non-essential tasks like stop-string checks whenever the GPU risks running dry.

Second, it dynamically discovers shared prefixes. For workloads that repeatedly query the same prompt or document, it runs a greedy search before each forward pass to group the longest common prefixes, cutting redundant attention compute via Hydragen.

Third, it tailors parallelism to your hardware. For nodes without NVLink, it provides a fast pipeline parallel setup. For high-end NVLink setups like H100 or B200 clusters, it leverages PyTorch Async Tensor Parallelism, automatically activating it once batch sizes cross 6,000 tokens to hide communication latency.

The engine is written in pure Python using FlashInfer kernels and currently supports single-node deployments for Llama-3 and Qwen-2 model families. You can grab the code on GitHub or install it directly via PyPI with pip install tokasaurus.

Why it matters

Anyone running offline synthetic data generation, RL rollouts, or massive batch jobs should test Tokasaurus on their workloads. It is not designed to power interactive customer chatbots, but for raw background crunching, the throughput jump translates directly into lower cloud bills.

What impresses us most is how seriously it treats non-NVLink hardware. Most modern inference systems assume expensive high-speed interconnects, leaving budget multi-GPU nodes bottlenecked on communication. Tokasaurus proves that disciplined pipeline scheduling can unlock massive throughput on everyday hardware, even if its model support is still far narrower than vLLM.