Nvidia's Open Speech Model Hits 5.33% Word Error in English With 4 Billion Active Parameters

Nvidia on Wednesday published Nemotron 3.5 Transcribe, an open-weight speech model that folds four jobs into one checkpoint steered by plain-language prompts: transcription, speaker attribution, translation and question answering about the audio. The weights are on Hugging Face and ModelScope as nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B in BF16, FP8 and NVFP4 formats.

The architecture is a 30-billion-parameter mixture of experts that activates roughly 4 billion parameters per token, pairing a speech encoder with a hybrid Mamba-Transformer language model. Nvidia's blog counts 30 billion parameters; the model card lists 34 billion including the audio encoder, of which 4.3 billion are active. That sparsity is what lets the model run on one GPU, from a Jetson AGX Thor controller or a DGX Spark desktop up to a Blackwell data-center card.

The headline numbers, all company-reported: 40 languages transcribed, 31 languages translated to and from English, speaker labels in 13 languages for up to eight speakers, recordings up to one hour, and word-level timestamps. On the Hugging Face Open ASR Leaderboard the model reports a 5.33% average word error rate for English against open-weight rivals, 4.62% mean WER across seven languages and 10.98% on long-form audio. Artificial Analysis separately measured 2.8% AA-WER, which ties Mistral's Voxtral Small for first among open-weight models while activating about 4 billion parameters against roughly 24 billion for the model it tied.

Speaker attribution is where the reported gap is widest. Nvidia cites an 18.26% diarization error rate across 21 languages on David AI's benchmark, the lowest of 14 systems tested, and first place at 32.4% concatenated minimum-permutation WER in VoiceArena's comparison. The model uses Sortformer speaker cues during generation, which is why speaker labels come out of the same pass as the transcript rather than a second pipeline stage.

The release ships alongside Granary v2: 463,062 hours of curated multilingual speech across 48 languages and 16 source corpora, up from 25 languages in the first version. Nvidia's NeMo Curator pipeline generates three transcript variants per recording, verbatim with disfluencies, inverse-text-normalized with disfluencies, and inverse-text-normalized without, plus translations with quality scores, contextual biasing terms and speech-grounded question-answer pairs.

The most concrete argument for adaptation comes from a case study on Eka Care's Indian medical speech benchmark. Out of the box the model erred on 30.8% of words; a prompt specifying English and Latin script cut that to 23.8%; fine-tuning on synthetic and real medical speech for about 14 GPU-hours on a GB300 brought held-out speaker WER down to 12.17%, a 45.8% relative reduction from the best prompt. The adapted model beat an open medical specialist model at 13.79% and a widely used general-purpose ASR system at 14.92%.

Throughput claims of this kind need independent testing. Nvidia says a single RTX PRO 6000 Blackwell Server Edition GPU at NVFP4 precision processes 100 hours of audio in under three minutes, measured on 30-second clips at high concurrency. Two limits matter operationally: the model is built for recorded audio and does not handle live streams, so real-time captioning is out, and serving is through vLLM or NeMo-Speech.cpp, with hosted endpoints from Baseten, DeepInfra and Simplismart.

The timing is not accidental. Mistral shipped Voxtral 3 this month, ElevenLabs launched Realtime for voice agents, and Salesforce released Launa-Omni. What Nvidia is selling is not the top of the leaderboard but the option to keep sensitive audio, from internal meetings to patient records and compliance calls, on hardware the customer already owns, at roughly 4 billion active parameters instead of 24 billion or more.