Inception Labs launches Mercury Coder to generate 1,000+ tokens per second with diffusion

If you've watched Midjourney or Sora generate media, you know the routine: the screen starts as pure static and snaps into focus through rapid denoising steps.

Text, on the other hand, has stubbornly stayed sequential. For years, language models have spit out words one by one from left to right. That autoregressive pipeline has dominated the field, and it has long been the primary bottleneck for inference speed.

Inception Labs, a startup founded by Stanford professor Stefano Ermon, is flipping that script. The company just launched Mercury Coder, billing it as the first commercial-scale diffusion language model (dLLM). Instead of predicting one token after another, it crafts entire code sequences in parallel, hitting over 1,000 tokens per second on commodity Nvidia H100 GPUs.

Ditching the typewriter approach

Traditional LLMs work like high-speed typewriters. To write the hundredth word of a script, the model must wait for the ninety-ninth word to exist.

Mercury Coder throws that sequential chain out the window. Starting from noise, the model applies a coarse-to-fine diffusion process, refining the whole block of code simultaneously across several iterations.

Because the entire sequence is generated at once, the model takes full advantage of GPU parallel processing. Inception Labs claims this method slashes inference costs by 10x while delivering speedups up to ten times faster than conventional approaches.

Early benchmarks show that Mercury Coder matches or outperforms speed-optimized stalwarts like OpenAI’s GPT-4o Mini and Anthropic’s Claude 3.5 Haiku, while performing similarly to Gemini 2.0 Flash-Lite on coding evaluations.

1,000 tokens a second on standard GPUs

Blasting past 1,000 tokens per second on code generation usually requires exotic, hyper-specialized inference hardware like Groq or Cerebras.

Mercury pulls this off on regular Nvidia H100s. AI researcher Andrej Karpathy weighed in on X, calling the shift an intriguing departure from industry habits. Karpathy noted that while image and video generation fully embraced diffusion, text has curiously resisted it for years. In his view, Mercury could reveal entirely new strengths and psychological quirks in AI text generation.

Inception Labs has opened a web playground at chat.inceptionlabs.ai where users can test code prompts alongside an interactive software preview. Enterprise tiers featuring Mercury Coder Mini and Small are available via API or on-premise setups, with conversational models already in closed beta.

Why it matters

We think Mercury Coder's biggest achievement is proving that text generation isn't married to the autoregressive paradigm.

For developers building coding copilots, customer support agents, and workflow automations, latency is everything. If you can serve ten times more requests on the exact same Nvidia hardware you already rent, the cost calculus for production AI shifts overnight.

It also poses a serious strategic question for the industry. Hardware startups have raised billions building bespoke silicon just to accelerate token-by-token generation. If algorithmic changes like diffusion can deliver those exact same real-time speeds on standard GPUs, the moat around specialized inference chips suddenly looks a lot shallower.

The real test, in our view, will be long-horizon logical consistency. Parallel denoising works brilliantly for modular code blocks and syntax patterns, but whether diffusion can maintain coherent reasoning across massive codebases remains to be seen. Still, Inception Labs just opened a viable second front in the AI architecture race.