Meta drops SAM Audio to segment sounds like images

Meta just brought its famous "click to cut out" magic to sound with SAM Audio, a new unified multimodal model for audio separation.

Much like the original Segment Anything Model did for computer vision, SAM Audio lets you isolate any individual sound from complex recordings using natural text, visual cues, or time stamps.

Point, click, and isolate any track

Audio editing has always been fragmented, forcing creators to bounce between dedicated vocal removers, noise reduction plugins, and stem separators. SAM Audio combines all of that into one unified generative framework.

Its biggest trick is multimodal prompting. If you have footage of a live band, you can simply click on the guitar in the video to isolate its audio track. You can also type text prompts to wipe out background traffic, or use span prompts to highlight a barking dog once and erase it from an entire podcast recording.

Under the hood, SAM Audio is powered by a flow-matching diffusion transformer. It takes mixed audio alongside your multimodal prompt, encodes them into a shared space, and outputs both the isolated target audio and the residual track.

Giving the model a set of ears

To make visual-audio alignment work, Meta also released Perception Encoder Audiovisual, or PE-AV. Think of PE-AV as the ears and SAM Audio as the brain. Built on the open source Perception Encoder launched in April, it extracts frame-level video features and aligns them with sound representations.

Alongside the models and research papers, Meta introduced SAM Audio-Bench, an in-the-wild separation benchmark, and SAM Audio Judge, an automated evaluation model. The whole setup is already live in Meta's Segment Anything Playground for anyone to test with uploads.

Why it matters

For podcasters, video editors, and mobile creators, this is an immediate game changer. Salvaging ruined audio usually takes meticulous cleanup or expensive specialized software. Replacing that entire workflow with a simple text prompt or a click on the screen makes high-grade audio cleanup accessible to anyone, and it is definitely worth trying in the playground today.

For the broader industry, Meta is applying its Segment Anything playbook to audio, threatening standalone stem separation and noise cancellation tools. By launching the benchmark and judge model at the same time, Meta is also angling to define the evaluation standards for the field.

We think grounding sound extraction in visual context is brilliant, as real-world audio rarely happens in a vacuum. The lingering question, however, is computational overhead. Running a flow-matching diffusion transformer locally on consumer devices or inside real-time editing software might prove far more demanding than legacy digital signal processing tools.