Twelve Labs takes on video understanding with Pegasus-1

The way most frontier models look at video today is surprisingly crude: they slice a clip into thousands of still frames, feed them one by one into an image model, and hope for the best.

That brute-force shortcut falls apart the second a video runs longer than a minute. Costs explode, latencies spike, and crucial context linking sound to motion gets lost along the way.

Now, AI research outfit Twelve Labs is pushing back with Pegasus-1, an 80B-parameter multimodal foundation model designed from scratch to handle video natively.

Video-first instead of image-first

Instead of treating moving pictures like a heavy photo album, Twelve Labs built Pegasus-1 around a dedicated three-tier pipeline. It combines a video encoder derived from their Marengo embedding model, a video-language alignment layer, and a large language model decoder.

The entire system clocks in at roughly 80 billion parameters, though Twelve Labs has not yet detailed how those weights are divided among the three components.

By processing visual, speech, and ambient audio signals simultaneously, Pegasus-1 claims state-of-the-art results on standard video summarization benchmarks with a single API call.

Alongside the model, Twelve Labs rolled out three specific endpoints for developers.

The Gist API works out of the box to generate snappy titles, core topics, and hashtag lists without requiring custom prompts.

The Summary API automatically breaks footage into structured chapters, highlights, and narrative recaps.

For more specialized workflows, the experimental Generate API lets developers pass natural language prompts to turn video context into custom deliverables, ranging from formatted executive summaries to creative text.

Why it matters

We think Twelve Labs is hitting a massive blind spot in the current generative AI wave. Most multimodal APIs brag about video comprehension, but they are essentially reading transcriptions or glancing at a handful of freeze-frames. When the action on screen happens without speech, those tools go completely blind.

For builders dealing with massive video archives, enterprise training sessions, long-form podcasts, or media monitoring, Pegasus-1 eliminates the need to stitch together separate transcription and vision pipelines. Having a single call produce reliable chapters and deep context is a massive operational win.

The open question is operational efficiency. At 80 billion parameters, native video inference is an immense compute sink. Twelve Labs will have to prove that its pricing and latency can compete once enterprise customers start throwing hundreds of continuous hours at the endpoint.

Still, this sets a clear marker: the era of faking video understanding through image sampling is ending, and general-purpose giants will soon have to match video-native architectures.