Google unveils T5Gemma 2 with multimodality and 128K context

T5Gemma 2 is Google's new family of open encoder-decoder models built on top of Gemma 3. It marks the first time an encoder-decoder architecture packs both multimodal vision capabilities and a massive 128K token context window.

If you need a lightweight, highly capable base model to fine-tune for specific on-device vision and text workflows, this is built directly for you.

What it can do

T5Gemma 2 processes images and text side by side, handling visual question answering and multimodal reasoning through a lightweight vision encoder.

By adopting Gemma 3's alternating local and global attention, context spans up to 128K tokens. The model also supports over 140 languages right out of the box.

Google released three compact sizes: a tiny 270M-270M version packing around 370M total parameters excluding the vision encoder, a 1B-1B version at roughly 1.7B parameters, and a 4B-4B version reaching about 7B parameters.

Across official benchmarks in coding, reasoning, and multimodal understanding, T5Gemma 2 consistently beats its corresponding Gemma 3 base counterparts, while demonstrating much stronger long-context retention.

How it saves parameters

Instead of just scaling up training tokens, Google made two smart architectural tweaks to shrink memory overhead.

First, it ties embeddings across the encoder and decoder. Sharing this vocabulary table cuts millions of redundant weights, which is crucial for making the 370M version fit into tiny memory footprints.

Second, it merges decoder attention. Rather than running separate self-attention and cross-attention blocks, T5Gemma 2 combines them into a single unified layer, reducing structural complexity and boosting parallel inference efficiency.

Constraints and availability

You cannot use this right away as a plug-and-play chat assistant. Google is only releasing pre-trained checkpoints, leaving post-training and instruction tuning to developers.

The benchmark numbers Google shared came from minimal supervised fine-tuning without reinforcement learning, purely to illustrate headroom.

The pre-trained weights are now live across multiple open model platforms, ready to be plugged into custom downstream pipelines.

Why it matters

Decoder-only models have dominated local AI, but tasks like document extraction, image querying, and classification often run faster and cheaper on encoder-decoder setups.

For developers building on-device apps, getting 128K context and multimodal reasoning inside a sub-billion parameter footprint is a rare win. You have to handle the fine-tuning yourself, but the efficiency payoff makes it well worth testing.