Meta drops Llama 4 with MoE and 10M context

Llama 4 Scout and Llama 4 Maverick are Meta's latest open-weight multimodal AI models. Instead of sticking with traditional dense designs, both make the full jump to a mixture-of-experts (MoE) architecture, running on just 17 billion active parameters while integrating text, image, and video comprehension directly into the core backbone.

The real showstopper here is context length. The smaller Scout model pushes context support to an unprecedented 10 million tokens, while the larger Maverick caps out at 1 million tokens for general-purpose workloads.

What they can do

Both models are natively multimodal thanks to early fusion, meaning text and vision tokens are pre-trained together across 200 languages and massive image-video datasets. They comfortably handle multi-image reasoning across up to 48 images in pre-training and 8 images in post-training.

Llama 4 Maverick is the heavy hitter, packing 400B total parameters across 128 routed experts. Meta claims it outpaces GPT-4o and Gemini 2.0 across reasoning, coding, and image understanding, while trading blows with DeepSeek V3.1 in coding tasks. It is fine-tuned for high-end chat, complex writing, and deep visual problem-solving.

Llama 4 Scout features 109B total parameters with 16 experts. Its killer feature is its 10M token context window, engineered through a hybrid architecture called iRoPE that interleaves layers without positional embeddings (NoPE) alongside chunked attention. It can ingest entire company code repositories or multi-year document archives in a single prompt.

Both models were distilled from Llama 4 Behemoth, an upcoming teacher model with nearly 2 trillion parameters and 288B active weights that Meta says outperforms Claude 3.7 Sonnet and GPT-4.5 on STEM benchmarks.

Specs and how to run them

Deploying these models depends on how you handle memory footprint. Even though only 17B parameters light up per token, all base weights must still live in VRAM.

For Llama 4 Scout, you can fit the entire model onto a single 80GB NVIDIA H100 GPU using on-the-fly 4-bit (Int4) quantization. Unquantized, it requires a multi-GPU setup.

For Llama 4 Maverick, Meta recommends a single 8-GPU NVIDIA H100 DGX host, running either in BF16 or FP8 format. Serving can be handled via Hugging Face Text Generation Inference (TGI) or vLLM.

Both models are released under the Llama 4 Community License. Model weights and instruct checkpoints are live on Hugging Face (requiring transformers v4.51.0 or higher) and llama.com.

Why it matters

For developers, Scout’s 10M context window practically eliminates the headache of brittle RAG pipelines for massive technical docs or legal discovery. Being able to run that capability on a single quantized H100 lowers the experimentation cost dramatically.

We think Meta's transition to MoE and early-fusion multimodality is the right call at the right time. It delivers frontier-level reasoning at an inference speed comparable to standard 17B models. While memory requirements will still keep Maverick out of reach for hobbyists with desktop setups, Scout delivers one of the most capable and practical open-weight engines available today.