Tsinghua's Falcon reads lips to isolate voices in noisy calls

Falcon is an open-source audio-visual target speaker extraction model from researchers at Tsinghua University. Instead of relying purely on noisy audio signals, it watches the speaker's mouth movements through a camera to cleanly isolate their voice from background chatter in real time.

Accepted to NeurIPS 2026, the project released its paper, model weights, and inference code publicly on GitHub and Hugging Face on October 23.

What it does and how it works

Traditional speech separation often chokes when multiple voices overlap or room acoustics get messy. Falcon solves this by matching visual lip movements captured by a webcam with the incoming microphone feed, filtering out every voice except the target speaker.

Under the hood, Falcon avoids the common performance pitfall of repeatedly looping small separators. Instead, it processes features in a single encoder-decoder pass using multi-scale time-frequency refinement. While it borrows a frozen 11.2M parameter visual backbone for lip reading, Falcon's own trainable audio-visual separation network requires just 0.5M parameters.

To keep nearby voices from leaking through, the team introduced an encoder-space contrastive loss during training. It suppresses interfering speech at zero extra compute cost during actual inference.

Benchmarks and running it locally

That architectural diet makes a huge difference in speed. Compared to the causal AV-TF-GridNet baseline, Falcon slashes trainable parameters by 91.9%, cuts computational cost by roughly 20x, and drops GPU latency by 11x. Even against Swift-Net-12, another 0.5M parameter alternative, Falcon requires 77% less computation and cuts GPU latency by 70%.

Across multiple benchmarks including VoxCeleb2, LRS2, LRS3, and the team's newly released realistic RealSSA dataset, Falcon achieves state-of-the-art causal separation scores.

Best of all, the researchers built a ready-to-run desktop application called Falcon Studio. It runs entirely on a laptop using Apple Silicon, NVIDIA GPUs, or basic CPUs without phoning home. It tracks your lips via your webcam, cleans your voice, and can route the result into Zoom or Tencent Meeting through a virtual audio device. A live demo is also hosted on Hugging Face Spaces for quick browser-based tests.

Why it matters

If you frequently take video calls from noisy coffee shops or shared offices, Falcon is well worth checking out. Audio-visual speech separation has spent years locked inside research labs because multi-modal architectures were far too heavy for personal devices. By shrinking the trainable core to 500,000 parameters and wiring it directly into virtual meeting inputs, Falcon turns a frontier research problem into an everyday productivity hack.

The real win here is practical edge deployment: developers working on smart glasses, robotic assistants, or voice agents now have an off-the-shelf way to do targeted listening on constrained chips. The obvious catch is line-of-sight: if your face moves off-camera or poor lighting obscures your mouth, the extraction quality will slip. But for standard video conferencing, it is easily one of the smartest lightweight audio tools to land on GitHub recently.