Google's VideoPoet trades diffusion for LLMs to generate video

If you have spent any time playing with AI video tools like Runway Gen-2, you know the frustration: as soon as a character makes a sudden move or runs across the frame, the whole thing tends to melt into a blurry, distorted mess. Most current video generators rely on diffusion models, and big, coherent motions remain their Achilles heel. Google Research just decided to tackle this problem from an entirely different angle: treating video as text.

On December 19, Google Research engineers Dan Kondratyuk and David Ross unveiled VideoPoet, a large language model designed from the ground up for zero-shot video generation. Instead of following the diffusion pack, VideoPoet uses an autoregressive LLM to churn out 10-second video clips packed with large, coherent motions and noticeably fewer artifacts.

Video, images, and audio as discrete tokens

Almost every headline-grabbing video model today relies on diffusion techniques. Google researchers went against the grain because LLMs have already proven to be monster multitaskers across text, code, and audio, and they can leverage years of scaling improvements built for language models.

To make visual frames readable by an LLM, VideoPoet relies on tokenizers. It uses MAGVIT V2 to encode video frames and still images into sequences of discrete integer tokens, and SoundStream to do the same for audio. Special task tokens and boundary tokens tell the model what job to perform.

Once everything is converted into tokens, VideoPoet operates just like GPT-style models predicting the next word, but here it predicts the next visual or acoustic token. Decoders then unpack those tokens back into viewable video and playable sound.

Multitasking inside a single architecture

Because everything speaks the same token language, Google did not need separate specialized models for each task. VideoPoet handles text-to-video, image-to-video animation, video inpainting, outpainting, video stylization, and even video-to-audio generation within a single system.

The demonstrations show complex prompts like a raccoon dancing in Times Square or a horse galloping through Van Gogh's Starry Night, maintaining coherent, wide-ranging motion over 10 full seconds. For stylization, the system uses optical flow and depth maps to re-skin existing video into entirely new artistic styles.

Why it matters

We think VideoPoet is a pivotal reality check for the generative video space, showing that diffusion is not the only viable road forward.

For creators and animators, the promise of an all-in-one architecture is huge. Right now, making a polished AI clip feels like juggling five disconnected apps: one for base images, another for video motion, another for inpainting, and an external tool for sound effects. VideoPoet puts all of that under one roof. The catch is that Google has only published the research and demo page, meaning creators cannot log in and test it in their actual workflow yet.

For the industry, VideoPoet puts serious pressure on startups like Runway and Pika. If autoregressive LLMs can genuinely solve the motion coherence problem, video generation will plug directly into the brutal scaling laws of large language models. The tech giants with the largest compute clusters could quickly outmuscle specialized diffusion models.

Still, Google's approach is not without trade-offs. Compressing high-definition continuous video into discrete tokens inevitably introduces fidelity loss compared to native latent diffusion. Autoregressive token generation for long, high-frame-rate clips is also notoriously expensive to run at scale. Google proved the architecture works, but making it fast and cheap enough for consumer software will be the real hurdle.