Google gives local LLMs a printing press upgrade
Google just open-sourced DiffusionGemma, an experimental 26B MoE model that ditches token-by-token generation for text diffusion, stamping out 256 tokens in parallel. It delivers up to 4x faster GPU inference, clocking over 1,000 tokens per second on an H100 and fitting into 18GB VRAM when quantized. Why it matters: It drastically cuts latency for local inline editing and code infilling, though Google notes raw output quality still trails standard Gemma 4.
DiffusionGemma is an experimental open model from Google that ditches standard token-by-token text generation in favor of text diffusion. Instead of tapping out words one by one like a typewriter, it stamps down entire 256-token text blocks at once, speeding up GPU text generation by up to 4x.
When running models locally for a single user, GPUs spend most of their time bottlenecked by memory bandwidth, waiting for each sequential token. DiffusionGemma shifts that workload straight into raw compute, acting much more like an industrial printing press.
How it works
Diffusion models usually live in image generation, starting with random static and refining it into a picture over multiple steps. DiffusionGemma brings that exact loop to text.
It begins with a canvas of 256 placeholder tokens. Over multiple parallel passes, the model locks in confident tokens and uses them as bidirectional context clues to iteratively correct mistakes across the entire block before converging on the final text.
Because all 256 tokens are processed simultaneously with bidirectional attention, every word sees both what comes before and what comes after. That makes it uniquely suited for non-linear tasks like inline text editing, code infilling, or mathematical structures. In fact, Unsloth fine-tuned DiffusionGemma to solve Sudoku puzzles, a task that trips up standard sequential models because numbers depend heavily on future choices.
Specs, speed, and limitations
DiffusionGemma is a 26B Mixture of Experts model, but it only activates 3.8B parameters during inference. Quantized, it fits comfortably into 18GB of VRAM on high-end consumer GPUs.
On the speed front, it clocks over 1,000 tokens per second on an enterprise NVIDIA H100 and pulls over 700 tokens per second on a desktop GeForce RTX 5090.
There is an obvious tradeoff: Google explicitly notes that DiffusionGemma's raw output quality trails standard Gemma 4. If your pipeline demands pristine, high-accuracy reasoning, standard Gemma 4 remains the recommendation. Furthermore, its massive throughput advantage shines primarily in single-user local setups with low batch sizes, high-concurrency cloud environments will see diminishing returns.
How to run it
Google released the experimental weights under an Apache 2.0 license on Hugging Face.
You can run it using Hugging Face Transformers, MLX on Apple Silicon, and vLLM, with llama.cpp support planned soon. For fine-tuning, it integrates with Unsloth, NVIDIA NeMo, and Google's JAX toolbox Hackable Diffusion. NVIDIA systems also get native NVFP4 kernel optimizations across RTX 4090, 5090, and enterprise hardware.
Why it matters
If you build local-first developer tools or instant-response interactive apps, DiffusionGemma is well worth exploring. Zero-latency inline code infilling and real-time grammar repair are notoriously hard to run smoothly on a local machine, and bidirectional diffusion tackles both without the usual sequential delay.
We think Google made a clever move here by releasing this as an unapologetic experimental utility. Instead of chasing benchmark leaderboards against frontier autoregressive models, it isolates a real engineering bottleneck (single-user local latency) and delivers a tangible 4x speedup right to consumer hardware. It proves that text diffusion is ready to leave the research lab and offer a genuine alternative for speed-critical workflows.
Sources (1)
- blog.google(opens in a new tab)Originalblog.google