Fudan and Tencent open-source Prism for native 2K video-audio generation

Prism is an open-source framework from Fudan University, Tencent Hunyuan, and Zhejiang University built for native 2K joint video-audio generation. It generates high-resolution video and tightly synchronized audio together in a single pipeline, scaling up to 1440x2560 without relying on separate audio post-processing.

Generating 2K video natively with conventional full attention is notoriously expensive. Computing attention across millions of visual and audio tokens incurs quadratic complexity, spreading GPU cycles thin over static backgrounds and silent pixels while muddying fine details.

Cutting redundant compute with dynamic sparse attention

Prism solves this bottleneck by introducing dynamic sparse attention, effectively teaching the model to focus only on where the action and sound actually happen.

Instead of treating every pixel equally, Prism groups tokens into spatiotemporal macro-zones. It monitors two key signals: the variance of visual features along the channel dimension, and the feature norms from audio-to-video cross-attention. This tells the system exactly which areas are undergoing rapid visual movement and which areas are driving the sound.

Prism then dynamically assigns tailored block shapes to each zone, applying dense, fine-grained computation only to regions with strong audio-visual coupling. By pruning irrelevant computation, Prism achieves a 2.5x training speedup over full attention while delivering sharper visuals and tighter sound alignment.

Specs, hardware, and getting started

The code, data preprocessing pipelines, and preview checkpoints are available on Hugging Face under FrancisRing/Prism. The preview release includes two weight variants, preview_alpha and preview_beta, built on top of a MOVA-360p base model.

Prism handles text-to-video-audio and image-to-video-audio workflows. Users can guide the generation with structured audio tags such as music, sfx, and speech embedded directly inside prompts.

Running Prism requires serious compute. The stack relies on Python 3.10+, PyTorch 2.7.1, CUDA 12.4, and FlashAttention. For 720p inference, you can get away with a single 80 GB GPU using CPU offloading and DeepSpeed. For native 1080p and 2K outputs, multi-GPU setups running FSDP with sequence parallelism are strongly recommended, alongside spatial VAE tiling to avoid out-of-memory errors.

Why it matters

Most video models still treat audio as an afterthought, generating silent footage first and patching in sound effects later. Prism proves that native, end-to-end 2K video-audio generation does not have to collapse under quadratic compute costs if you route attention dynamically based on cross-modal activity.

That said, Prism is primarily aimed at research labs and well-funded production pipelines rather than desktop hobbyists right now. An 80 GB GPU floor for entry-level inference means individual creators will have to wait for distilled or aggressively quantized community forks before running native 2K sound-and-video on local hardware.