Nvidia's new open speech model takes natural language prompts

Nvidia just open-sourced Nemotron 3.5 Transcribe, a unified speech model that bundles transcription, speaker diarization, translation, and audio question answering into a single checkpoint.

What it can do

Nemotron 3.5 Transcribe handles recordings up to an hour long and can distinguish up to eight different speakers. It supports transcription across 40 languages, translates between 31 languages and English, and attributes speakers in 13 languages.

What makes it genuinely useful is prompt steering. Instead of fixing output styles via rigid post-processing regex, you can use natural language prompts to ask for verbatim text with all the ums and uhs preserved, or demand a clean, normalized transcript. You can feed it domain-specific vocabulary like medical jargon before transcribing a doctor-patient conversation, or ask it directly what decisions were made in a meeting. Even in chaotic screwball-dialogue scenes where people constantly talk over one another, it manages to track turns and word-level timestamps.

Under the hood and specs

Architecturally, Transcribe combines a speech encoder with a hybrid Mamba-Transformer mixture-of-experts language model. While the model has around 30B total parameters, only about 4B active parameters run per token. That architecture places it on the accuracy-efficiency Pareto frontier, tying for first place on Artificial Analysis's speech leaderboard at a 2.8% error rate while running at one-sixth the active footprint of rivals. It also took first place among open-weight models on the Hugging Face Open ASR Leaderboard with a 5.33% English word error rate, and posted the lowest diarization error rate across 21 languages on David AI's benchmark.

For deployment, the full BF16 checkpoint weighs around 67.5GB, while the upcoming NVFP4 quantized version shrinks under 22GB. On a single RTX PRO 6000 Blackwell Server Edition GPU, it clocked 100 hours of audio transcribed in under three minutes. You can serve it today via vLLM, or run it locally using NeMo-Speech.cpp on hardware ranging from Jetson AGX Thor to RTX Spark workstations. Note that it is built specifically for recorded audio, not real-time streaming.

Alongside the weights, Nvidia released Granary v2 on Hugging Face and ModelScope, sharing the four-stage curation pipeline and public manifests for over 463,000 hours of multilingual audio.

Why it matters

Until now, pulling off reliable meeting transcription required chaining separate systems together: Whisper for text, Pyannote for speaker diarization, and an LLM for formatting or summarizing. That meant high latency, complex pipelines, and compounding errors.

Nvidia effectively collapses that entire stack into one open-weight model you can steer with natural English. For developers building meeting summarizers, podcast tools, or offline medical dictation apps, being able to run an all-in-one FP4 model on a single GPU makes this the new open-source standard to beat.