xAI rewrites Grok-2 inference stack, doubling Grok-2 mini speed

Just days after releasing Grok-2, Elon Musk's xAI did something unusual: it tore down the model's entire inference stack and rebuilt it over a single weekend.

According to xAI developer Igor Babuschkin, two team members—Lianmin Zheng and Saeed Maleki—completely rewrote the company's serving code from scratch in three days. They swapped their previous setup for SGLang, an open-source serving framework.

The result was immediate. Grok-2 mini is now running twice as fast as it was yesterday. The architectural overhaul also made the full-size Grok-2, which requires multi-host inference across multiple machines, fast enough to serve at reasonable speeds. Babuschkin added that both models didn't just get faster, but also slightly more accurate.

Open source fixes the latency gap

The engine behind this sudden speed jump is SGLang, an Apache 2.0-licensed system created by researchers from Stanford, UC Berkeley, Texas A&M, and Shanghai Jiao Tong University.

SGLang connects a frontend language with an optimized backend runtime. By automatically reusing key-value caches and executing parallel requests inside a single program, the system claims up to 6.4 times higher throughput than conventional serving frameworks. It already supports models like Llama, Mistral, and LLaVA, as well as API services like OpenAI's GPT-4.

Meanwhile, Grok-2 is holding its own on the benchmarks. On the LMSYS Chatbot Arena leaderboard, Grok-2 secured the number two spot globally with a score of 1293 across 6686 votes. That ties it with Google's Gemini-1.5 Pro and puts it just behind OpenAI's latest ChatGPT-4o, while leading the May 2024 build of GPT-4o. Grok-2 also claimed first place specifically in mathematics tasks.

Grok-2 mini climbed to fifth overall with an arena score of 1268, sitting right behind GPT-4o mini and Claude 3.5 Sonnet. Both models remain exclusive to X subscribers paying the $8 monthly fee.

Why it matters

For paying X users, this is an instant quality-of-life upgrade. Token generation speed makes or breaks conversational AI, and Grok-2 mini doubling its output rate turns it into a genuinely responsive daily assistant for quick drafts and research. If you pay the $8 subscription, the speed jump makes the mini version worth a fresh spin today.

For the AI industry, the real story is how quickly modern open-source tooling can outpace custom proprietary infrastructure. Two engineers rewrote a core production stack in 72 hours by pulling SGLang off the shelf. We think this shows two things: academic open-source systems are now enterprise-ready out of the gate, and xAI's culture operates closer to a high-stakes hackathon than a cautious enterprise lab.

The flip side is that it reveals how rough Grok-2's initial deployment must have been. Serving a multi-host frontier model at poor latency was clearly an emergency fix. While xAI solved the immediate bottleneck cleverly, keeping pace with OpenAI and Google will require turning these ad-hoc weekend sprints into long-term architectural stability.