VidEoMT turns plain ViT into a video segmenter at 160 FPS

VidEoMT is a lightweight, encoder-only video segmentation model that cuts out dedicated tracking components and turns standard Vision Transformers into fast, reliable object trackers.

It handles challenging online video tracking across complex scenes and rapid motion, cleanly segmenting objects like people, sheep, and zebras across frames. Running with a standard ViT-L backbone, VidEoMT hits speeds of up to 160 FPS—delivering a 5x to 10x speedup over many existing online segmentation setups while preserving competitive accuracy.

How it works

Most existing online video segmentation pipelines duct-tape a per-frame segmenter to a specialized, heavy tracking module. While effective, that extra machinery piles on architectural complexity and eats up GPU cycles.

VidEoMT takes inspiration from the fact that sufficiently scaled and pre-trained plain ViT encoders can already handle precise segmentation on their own. Instead of bolting on an external tracker, the model introduces a lightweight query propagation mechanism that simply reuses object queries from the previous frame to follow targets through time.

To keep the model from getting blindsided by new subjects or sudden scene shifts, VidEoMT blends those propagated queries with a set of temporally-agnostic learned queries. This simple fusion gives the network tracking capabilities almost for free, avoiding dedicated tracking overhead altogether.

How to run it

The research paper and open-source code are available, along with instructions for setting it up and running tests locally using a ViT-L backbone.

Why it matters

If you build real-time vision pipelines, robotics, or video editing tools, VidEoMT is well worth a look. High-accuracy video object segmentation is notorious for bogging down systems, but hitting up to 160 FPS opens practical headroom for real-time edge applications without expensive multi-model setups.

We think the smart move here is doing away with architectural bloat. VidEoMT proves that foundation vision encoders often have temporal tracking hidden under the hood, and a clean query propagation trick is enough to unlock it.