IBM blends Mamba and Transformer in new Granite 4 models

IBM's newly open-sourced Granite 4 is a family of language models that merges the Transformer's attention mechanism with Mamba-2's state space architecture, designed specifically to slash the massive memory footprint of long-prompt AI inference.

The lineup spans four models from 3 billion to 32 billion parameters, offering developers a viable way to cut hardware costs without sacrificing output quality.

The lineup and how it works

Granite 4 features four models tailored for different workloads.

The smallest, Granite-4.0-Micro, has 3 billion parameters and sticks with standard Transformer attention to prioritize crucial tokens in text.

The other three models go hybrid. Granite-4.0-H-Micro features 3 billion parameters, while Granite-4.0-H-Tiny comes in at 7 billion parameters. Both are aimed at latency-sensitive use cases that prioritize raw speed.

The flagship of the family is Granite-4.0-H-Small at 32 billion parameters. It relies on a mixture-of-experts (MoE) design that activates only 9 billion parameters to answer a prompt, making it suitable for tasks like automated customer support.

The core trick lies in Mamba-2, a state space architecture originally derived from spacecraft trajectory calculations. When prompt lengths double, standard Transformers quadruple their RAM usage. Mamba trims that requirement down to a fraction, and Mamba-2 compresses one of the core components into about 25 lines of code for leaner execution.

In IBM's internal benchmarks, the 7B Granite-4.0-H-Tiny required just 15GB of RAM—one-sixth of what its previous-generation Granite 3.3 8B model needed—while delivering improved response quality. IBM noted that while the hybrid design drives efficiency, accuracy gains mostly stem from revamped training, post-training methods, and cleaner data.

Where to find it

Granite 4 is already available on IBM's watsonx.ai and Hugging Face, alongside half a dozen other third-party platforms.

IBM plans to bring the models to Amazon SageMaker JumpStart and Microsoft Azure AI down the road, and teased future variants focused on advanced reasoning.

Why it matters

If you run on-premise infrastructure or build latency-critical bots on tight compute budgets, Granite 4 deserves a spot in your test pipeline.

The Transformer's quadratic memory growth has always been a headache for production long-context tasks. While researchers have praised Mamba for months, big enterprise players rarely package it into ready-to-deploy open-source weights. Squeezing a 7B model down to 15GB of RAM makes deployment on accessible hardware realistic, offering developers a highly pragmatic escape hatch from runaway inference bills.