Nvidia's new Nemotron Ultra takes on DeepSeek R1 at half the size

Llama-3.1-Nemotron-Ultra-253B-v1 is Nvidia's new open-source 253-billion-parameter LLM built to rival massive reasoning models while fitting into significantly tighter hardware setups.

It features a built-in reasoning switch that lets developers toggle between straightforward, fast outputs and deep-thinking chain-of-thought processing directly via system prompts.

How Nvidia trimmed the footprint

Instead of training from scratch, Nvidia took Meta's older Llama-3.1-405B-Instruct and ran it through neural architecture search (NAS).

The team stripped out redundancies by introducing skipped attention layers, fused feedforward networks (FFNs), and flexible FFN compression ratios. The structural overhaul shrunk the dense model down to 253B parameters without wrecking output quality, making it fit on a single 8x H100 GPU server.

From there, Nvidia put the model through a heavy post-training pipeline. It absorbed 65 billion tokens during a knowledge distillation phase, followed by another 88 billion tokens of continual pretraining across datasets like FineWeb, Buzz-V1.2, and Dolma.

To lock in the reasoning behavior, they applied supervised fine-tuning across code, math, and tool-use, capped off with Group Relative Policy Optimization (GRPO) reinforcement learning.

Real-world performance and hardware specs

Flipping the reasoning switch unleashes dramatic jumps across benchmarks. On MATH500, accuracy climbs from 80.40% in standard mode to 97.00%. On the tougher AIME25 math test, it surges from 16.67% to 72.50%, while LiveCodeBench coding scores jump from 29.03% to 66.31%.

When stacked against DeepSeek R1—a massive 671B parameter Mixture-of-Experts model—Nvidia's 253B dense model holds its ground and frequently pulls ahead. Nemotron beats DeepSeek R1 on GPQA general QA (76.01 vs. 71.5), IFEval instruction following (89.45 vs. 83.3), and LiveCodeBench (66.31 vs. 65.9).

DeepSeek R1 still keeps the upper hand in raw competition math, leading AIME25 at 79.8 against Nemotron's 72.50.

The model supports a 128k context window and has been validated for Nvidia Hopper and B100 architectures in both BF16 and FP8 precision modes. Weights, inference code, and post-training data are fully open on Hugging Face, running natively on Transformers 4.48.3 or newer.

Why it matters

For enterprise teams and self-hosters, running DeepSeek R1 has meant managing sprawling multi-node MoE infrastructure. Being able to run comparable reasoning and superior coding on a single 8-GPU box dramatically lowers the bar for on-prem agent workflows.

We think this release showcases the true leverage in post-training and architectural distillation. Nvidia didn't try to win a parameter arms race; it engineered an efficient sweet spot that turns an older open model into a lean, production-ready powerhouse.