Google drops Gemma 3 with multimodal powers

Gemma 3 is Google's newest open-weight model family, available in four sizes: 1B, 4B, 12B, and 27B parameters. While the lightweight 1B remains text-only, the larger three variants process both images and text with a massive 128k token context window, covering over 140 languages.

What it does

Gemma 3 handles complex document summarization, visual QA, and multilingual reasoning. The 1B variant expands its context window from 8k to 32k tokens, while the 4B, 12B, and 27B models scale all the way to 128k.

Google released both base pre-trained and instruction-tuned versions on Hugging Face. When working strictly with text, developers can opt not to load the vision encoder into memory, preserving valuable GPU memory.

The benchmark gains are dramatic. The 4B instruction-tuned model outperforms the previous generation's Gemma 2 27B, while the new 27B instruction-tuned variant surpasses the proprietary Gemini 1.5 Pro across multiple standard benchmarks.

How it works

Instead of wasting compute pre-training 128k sequences from scratch, Google pretrained models on 32k lengths and only extended the 4B, 12B, and 27B variants to 128k near the end of training. The RoPE base frequency was raised from 10k to 1M and scaled by a factor of eight.

Memory overhead is managed through an interleaved sliding-window attention pattern. By interleaving five local layers with one global layer and cutting window size down to 1024 tokens, KV cache footprint drops significantly without hurting perplexity.

For visual inputs, Gemma 3 uses a SigLIP vision encoder operating on 896x896 crops. An adaptive pan-and-scan algorithm handles high-resolution and non-square images by cropping details before encoding. Image tokens receive bidirectional attention while text maintains causal attention. The model also doubles multilingual training data and adopts the Gemini 2.0 tokenizer.

All weights and an interactive Gemma 3 Space demo are live on Hugging Face.

Why it matters

If you build on-device agents or run local AI workflows, Gemma 3 is an immediate must-try. The fact that a 4B model beats the previous 27B generation means you can run powerful multimodal capabilities directly on laptops or single consumer GPUs.

At the top end, beating Gemini 1.5 Pro with an open 27B model narrows the gap between open-weight efficiency and closed enterprise APIs. Google is setting a high bar for accessible multimodal AI.