Google gives Gemma 4 a 3x speed boost

Google just released Multi-Token Prediction (MTP) drafter models for its Gemma 4 family, giving developers an easy way to speed up inference by up to 3x with zero loss in output quality. If you run Gemma 4 locally on consumer GPUs, Apple Silicon, or mobile devices, this is basically free performance.

Standard autoregressive generation bottlenecked by memory bandwidth forces heavy models to reload billions of parameters just to predict an obvious continuation. These lightweight drafters tackle that bottleneck head-on using speculative decoding.

How it actually works

Instead of letting a massive model slog through tokens one by one, a tiny MTP drafter model steps in and predicts a chunk of future tokens in parallel. Because the drafter is small, it finishes this draft in less time than it takes the target model to calculate a single token.

The larger target model then verifies all suggested tokens in a single forward pass. If it approves the sequence, it accepts the entire batch at once and generates one additional token of its own in the exact same step.

Crucially, output quality does not degrade at all because the primary Gemma 4 model retains absolute verification power. Under the hood, the drafters share the target model's KV cache and reuse activations, eliminating redundant computation. For edge-focused E2B and E4B models, Google also added an embedder clustering technique to clear vocabulary calculation bottlenecks.

Models and hardware gains

The drafters cover Gemma 4 31B Dense, the 26B MoE, as well as the lightweight E2B and E4B variants. On an NVIDIA RTX PRO 6000 running the 26B model, generation wait times were cut in half with overall speedups reaching up to 3x.

On Apple Silicon, running the 26B MoE at batch sizes between 4 and 8 unlocks roughly 2.2x faster speeds, with similar gains showing up on Nvidia A100 setups. Higher throughput on edge devices also means models finish faster and burn less battery.

The MTP drafters are open-sourced under the Apache 2.0 license. Weights are live on Hugging Face and Kaggle, with out-of-the-box support across vLLM, Ollama, MLX, SGLang, Transformers, and LiteRT-LM, plus quick mobile testing via Google AI Edge Gallery.

Why it matters

Memory bandwidth is the ultimate wall for local LLM deployment, and Google just handed everyone a painless workaround. You do not have to train, distill, or align a separate speculative model yourself; these official drafters plug straight into existing pipelines like Ollama and vLLM without sacrificing a single drop of reasoning capability.

We think this cements speculative decoding as a standard deployment pattern rather than an academic curiosity. If you are building local coding assistants, multi-step agent workflows, or snappy mobile apps on top of Gemma 4, this is an immediate, no-brainer upgrade.