Cerebras drops the fastest AI inference on Earth

Watching an AI model generate text often feels like waiting for someone to finish typing across an old chat app. Cerebras just decided to blow that experience wide open.

Chipmaker Cerebras Systems officially launched Cerebras Inference, a cloud service dedicated to running Meta's open-source Llama 3.1 models. The headline numbers are staggering: it pumps out 1,800 tokens per second on Llama 3.1 8B, and 450 tokens per second on the 70B variant.

According to independent benchmarks from Artificial Analysis, which clocked 8B at over 1,800 tokens/sec and 70B at over 446 tokens/sec, that makes it roughly 20 times faster than Nvidia GPU instances hosted on hyperscale clouds.

It is also cheap. The developer tier starts at 10 cents per million tokens for Llama 3.1 8B and 60 cents per million tokens for 70B, which Cerebras claims is up to 100 times better price-performance than rival GPU setups.

Bypassing the memory wall with a giant dinner-plate chip

Most modern GPUs struggle with inference not because their compute cores are slow, but because they cannot shove model weights back and forth across external memory fast enough.

Cerebras solves this problem with its CS-3 system powered by the Wafer Scale Engine 3 (WSE-3). Instead of cutting silicon wafers into hundreds of tiny chips, Cerebras uses the entire wafer as one massive processor. The result is 7,000 times more memory bandwidth than an Nvidia H100.

Crucially, Cerebras did not resort to extreme model quantization to hit these speeds. Artificial Analysis verified that both models run in native 16-bit precision throughout the entire pipeline, matching Meta's official output quality.

Integration is straightforward since the API is fully compatible with OpenAI's Chat Completions endpoint. Cerebras rolled out three tiers: Free, Developer (serverless pay-as-you-go), and Enterprise (custom SLAs and on-premise deployments).

Industry figures quickly took notice. Andrew Ng pointed out that multi-step agentic workflows that require chaining dozens of LLM calls will directly benefit from this speed. Search engine Perplexity and voice infra company LiveKit are also partnering to tap into the low-latency pipelines.

Why it matters

In our view, hitting 1,800 tokens per second is not just about making chatbots feel snappier. It fundamentally unlocks agentic software.

When models run this fast, multi-agent chains that draft, test, and critique code internally over 20 iterations can finish in two seconds instead of two minutes. For voice interfaces and real-time search, latency drops below the human threshold of noticeable delay. Developers building agent workflows or audio agents should test this immediately, especially given the OpenAI API compatibility and low pricing.

There are real questions about scale, however. A single wafer-scale chip is an engineering marvel, but manufacturing and packaging them is notoriously difficult. Whether Cerebras can deliver enough cluster capacity to handle heavy enterprise spikes without latency degrading remains to be seen.

Even so, this is a brilliant tactical move. Inference already accounts for roughly 40% of the entire AI hardware market and is growing much faster than training. By targeting inference speed and memory bandwidth instead of fighting Nvidia's dominance in training clusters, Cerebras is proving there is plenty of room for specialized silicon to disrupt the status quo.