Nvidia's PixelUMM kills the VAE for raw pixels

PixelUMM is an open-source multimodal model from Nvidia and the University of Waterloo that handles both image and video understanding and generation within a single decoder-only Transformer. Its key move is discarding traditional VAEs and external vision encoders, reading and writing raw pixels directly.

What it does

Built on a Qwen3-8B backbone, PixelUMM performs four core tasks: text-to-image, text-to-video, image understanding, and video understanding.

Instead of compressing inputs into a latent space, it cuts images into 16x16 patches and groups video frames into 4-frame tubes. Inside the Transformer, separate understanding and generation experts share a single self-attention mechanism. Clean conditioning data like text prompts feeds into the understanding expert, while noisy target tokens flow into the generation expert.

Nvidia released two checkpoints. The default S8-F22-R05 supports all four tasks. An intermediate checkpoint, S8-F18-R01, drops video understanding but produces slightly cleaner video generations at 480p and 720p settings.

Output quality is still behind frontier commercial generators. Because the model uses a linear pixel projection head, generated images and videos can display subtle grid-aligned patch artifacts, particularly across smooth areas like skies.

How to run it

The code is live on GitHub under an Apache-2.0 license, while certain DiT-derived files retain CC BY-NC 4.0 terms.

Model weights can be downloaded from Hugging Face at nvidia/PixelUMM alongside the standard Qwen3-8B tokenizer files. Text-to-video commands run safety checks by default, requiring gated access approval for nvidia/Cosmos-1.0-Guardrail.

Single-GPU hardware can handle inference, and the repo includes a check_checkpoint.py utility to verify checkpoint integrity before loading weights onto a GPU. However, running the repo's toy training workflow requires serious iron: seven GPUs with at least 48 GiB of VRAM each, plus disk space for an approximately 61 GB checkpoint.

Why it matters

AI pipelines have long relied on a fragmented setup, using vision encoders for understanding and VAEs for generative diffusion. PixelUMM proves that a single decoder-only model can bypass latent compression entirely and operate natively on raw pixels across all four multimodal tasks.

We see it primarily as an architectural preview rather than a drop-in tool for creators. The fidelity and consistency cannot challenge commercial models yet, but for developers exploring unified, encoder-free multimodal setups, it offers a solid and transparent codebase.