OpenAI's custom chip is here, and it's fast

The frontier lab that made GPUs scarce is now building its own silicone to run them.

At the Hot Chips conference on Tuesday, OpenAI shared the first benchmark results for Jalapeño, its first custom AI inference chip developed in partnership with Broadcom. The numbers suggest that OpenAI is ready to tackle one of the most stubborn trade-offs in AI hardware: choosing between high throughput and low latency.

Tested on SemiAnalysis's InferenceX benchmark, Jalapeño went head-to-head with Nvidia's GB200 and GB300 systems across three open-weight models: GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. The chip delivered 1.5 to 1.9 times more AI work per watt at peak throughput, cut end-to-end latency by 1.7 to 3.6 times, and posted 2.1 to 4.1 times higher performance in highly interactive workloads.

Designed by AI to kill the memory bottleneck

Inference has two distinct pain points. Prefill, which digests prompts, is compute-heavy. Decode, which spits out tokens one by one, is strangled by memory bandwidth and communication lag between chips. Jalapeño tackles this by keeping model state local, specifically the KV cache, across a large networking domain to eliminate idle cores waiting on data transfers.

While Jalapeño is rated at 700 watts, measured sustained power stayed at or below 550 watts during testing. On Kimi K2.5, the largest 1T parameter model tested, it achieved 3.4 times lower latency than the comparison system.

The development cycle was just as notable: OpenAI used its own frontier models to accelerate design and circuit optimization, shrinking the journey from concept to tapeout to just nine months. Later, using Codex with GPT-Astra, the team ported the three open models in two months. For select attention and mixture-of-experts blocks, AI-written kernels actually ran 1.5 to 1.8 times faster than human-written code.

Richard Ho, OpenAI's head of hardware, noted that Jalapeño will begin deploying in OpenAI's data centers by the end of this year in very small volumes, with significant deployment rolling into 2027. Gen 2 is already in development, though Ho emphasized that OpenAI will keep deploying accelerators from Nvidia and other partners.

Why it matters

We think Jalapeño is fundamentally about making long-horizon AI agents economically viable. When an agent has to reason through a ten-step workflow sequentially, small latency delays and power costs compound brutally. By dramatically lowering token latency without tanking power efficiency, OpenAI is trying to build a system where complex agentic loops do not bankrupt the balance sheet.

Strategically, this gives OpenAI tremendous leverage. Google has its TPU and AWS has Trainium; OpenAI entering the custom silicon arena with working hardware means it is no longer hostage to Nvidia's pricing cycles and allocation queues. Even if Nvidia's roadmap leaps forward before Jalapeño scales widely in 2027, having a viable first-party chip sets a hard ceiling on what OpenAI has to pay third parties.

It looks like a very calculated tactical choice: avoid the brutal complexities of frontier training silicon for now, and laser-focus on inference where daily serving costs live. The recursive loop of using AI models to co-design and program AI chips is no longer theoretical, it is actively running in silicon.