JD open-sources JoyAI-Video-Edit for real-time 720p video editing

JoyAI-Video-Edit is a new open-source system that lets you edit video streams in real time using natural language, modifying frames causally as they arrive without waiting for the full video to finish.

You can point your webcam at yourself or upload an existing clip, type an edit prompt, and watch the video transform on the fly. You can swap outfits, turn a room into a British castle, or paint out unwanted objects while the model seamlessly fills in the background.

What it can do

Traditional AI video editing processes clips offline in batches, meaning you upload a file, wait minutes for rendering, and hope the prompt hit the mark. JoyAI-Video-Edit treats video editing as an open-ended streaming task with around a 1-second operational latency.

In live demos, users can swap wardrobe styles continuously while moving in front of a live camera without breaking stride. The system supports full-scene stylistic shifts, such as turning footage into watercolor, as well as granular local edits like turning every dog white, adding party hats, and making sunglasses hot pink.

It also handles reference-image-guided video editing, or RV2V. You can pass a reference picture of a jacket or dress, and the model transfers that garment onto the subject in the video while keeping identity and clothing consistent across movement.

How to run it

Under the hood, the architecture stacks an MLLM condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion transformer (MMDiT). It relies on autoregressive distribution matching distillation and bounded KV-state inference to avoid runaway temporal drift and maintain stable per-chunk compute.

Throughput is its strongest card. In deployment benchmarks, the pipeline hits 30 FPS at 720 × 1248 resolution. For local desktop hardware, a single GeForce RTX 5090 with 32 GB of VRAM can sustain 840 × 480 streaming at 24 FPS. An RTX PRO 6000 reaches 24 FPS at 480p or 16 FPS at 720p.

The trade-off is memory. The model checkpoint weighs around 32.5 GB, so high-end GPU hardware is non-negotiable. Setup requires Python 3.10, compiling custom in-tree CUDA ops, and configuring SageAttention or flash-attn-4.

The project is fully open under the permissive Apache 2.0 license, meaning commercial deployment is welcome. Weights are hosted on Hugging Face alongside a live browser webcam demo, and the team has also open-sourced example training scripts and their LongV2VBench evaluation suite.

Why it matters

Most video diffusion tools still feel like rendering farm jobs. By pushing 720p diffusion editing to 30 FPS with negligible turnaround, JoyAI-Video-Edit shifts generative video into the realm of interactive camera filters and live virtual production.

If you build interactive video apps, live-streaming tools, or virtual wardrobe tech, this repo is well worth cloning today. While the 32 GB VRAM requirement keeps it out of reach for casual consumer hardware, proving that 16B diffusion models can edit live streams at full framerate sets a new bar for what open video infrastructure looks like.