Nvidia drops NVLM 1.0, an open rival to GPT-4o
Nvidia just released the NVLM 1.0 family of multimodal models, making weights public for its 72B flagship NVLM-D-72B. The model matches frontier proprietary systems like GPT-4o across vision tasks and unexpectedly gained 4.3 points in text benchmarks post-multimodal training, giving open-source developers frontier-grade power.
Nvidia just made a massive open-source play with the release of NVLM 1.0, a new family of multimodal models headlined by the 72-billion-parameter NVLM-D-72B. The model takes in both image and text inputs, rivals frontier systems like GPT-4o in vision-language tasks, and has its weights openly accessible right now.
Understanding memes, math, and dense visuals
Frontier vision capabilities have largely been locked behind proprietary APIs. NVLM-D-72B challenges that dynamic directly by using an architectural hybrid approach that blends multiple multimodal processing techniques.
In practical evaluations, the flagship model handles complex step-by-step mathematical reasoning from visual diagrams and can even break down nuanced academic memes. Across standard vision and language benchmarks, NVLM-D is competitive with the likes of GPT-4o, Claude 3.5, and Llama 3-V.
Gaining text prowess after vision training
Most multimodal models suffer from a form of catastrophic forgetting known as the "multimodal tax," where text and coding skills take a hit once visual encoders are wired in. NVLM-D-72B does the opposite.
Following multimodal training, the model's accuracy across key text benchmarks actually gained an average of 4.3 points over its original text backbone. The leap in pure text math and coding was sharp enough that independent researchers noted a 72B model performing on par with Llama 3.1 405B on reasoning evals, all while retaining strong vision capabilities.
Nvidia has published the research paper on arXiv, released the model weights to the public, and promised that the full training code will follow.
Why it matters
If you are an engineer or researcher looking for a frontier-class multimodal model that can run locally on an enterprise workstation without streaming data to proprietary endpoints, NVLM-D-72B should be at the top of your test list.
More broadly, Nvidia open-sourcing a model of this caliber is a serious shot across the bow of closed-source labs. By proving that multimodal training can actually sharpen text and coding performance rather than degrade it, NVLM sets a fresh technical bar that open-source builders and commercial competitors alike will have to reckon with.
Sources (1)
- venturebeat.com(opens in a new tab)Originalventurebeat.com