Xiaomi's open-source model generates full audio scenes

MiDashengLM-Gen is an open-source framework by Xiaomi researchers that generates a complete audio scene from a single text prompt. Instead of making you generate voiceovers, hunt down sound effects, and loop background tracks on separate editing tracks, it synthesizes speech, background music, foley, and ambient noise simultaneously into unified 16 kHz mono audio.

What it can do

Input relies on structured multi-view captions. You define the overarching scene, exact speech transcript, vocal characteristics and emotion, sound effects, music styling, and environmental acoustics. If an element is not needed, you simply flag it as unknown.

In public demos, the model handles complex scene dynamics cleanly. Examples range from a stand-up comedian delivering a punchline followed by roaring crowd laughter and a brass jazz sting, to a male speaker explaining car mechanics over an idling engine noise, or a female lawyer discussing business with instrumental underscore.

It also handles multi-language output and rich emotional delivery, such as Chinese speech delivered with palpable anger and frustration.

Keep expectations realistic on standalone music quality: this is not a dedicated music foundation model, and pure tracks will not match best-in-class open-source audio generators. But generating coherent layered audio with balanced speech and background layers in one pass is the real utility here.

How it works and hardware requirements

The pipeline pairs an autoregressive LLM backbone with Flow Matching. It extracts 768-dimensional latents via DashengTokenizer at 25 Hz, downsamples them to 5 Hz with an audio projector, and passes tokens into a fully fine-tuned Qwen3-1.7B.

Generation is driven by a 16-layer Flow Matching DiT using a 10-step Euler ODE solver, while a learned binary stop head automatically truncates the clip at variable lengths.

The deployment footprint is refreshingly lean. Total model files weigh in at under 12 GB, meaning you can comfortably run inference locally on a mainstream mid-range GPU. The code and model weights are released under the Apache 2.0 license, with full setup scripts hosted on GitHub.

Why it matters

For indie game developers, animators, and digital storytellers, this changes the baseline of rapid audio prototyping. Instead of juggling fragmented stock libraries and stitching three different AI generators together, you can draft an entire environmental soundscape from a structured prompt.

We think Xiaomi made a smart trade-off here. By anchoring the system to an efficient 1.7B language backbone and keeping the memory footprint under 12 GB, they prioritized multimodal integration and consumer hardware practicality over raw, ultra-high-fidelity music synthesis. It points toward a future where audio foley is generated as naturally as conversational text.