Tencent open-sources AuK, an audio model that edits speech like text

AuK is Tencent's open-source 1.5B foundation model that lets you edit spoken audio just like rewriting sentences in a text document, using simple natural-language prompts.

Instead of juggling separate noise-reduction plugins and voice cloning tools, AuK unifies speech generation and fine-grained audio editing into a single prompt interface. Tencent released both the full code and weights, offering a high-fidelity base model alongside AuK-Flash, a distilled variant built for ultra-fast 4-step inference.

What you can do with it

Most modern voice models can only synthesize new speech from scratch. AuK focuses heavily on in-place audio manipulation across several core scenarios.

First is content editing. If a speaker made a slip of the tongue, you do not need to re-record the whole take. You can simply feed the audio into AuK and prompt it to swap a phrase, drop an unnecessary sentence, or insert new lines. The model splices the new words in while preserving the original voice, cadence, and acoustics.

Second is paralinguistic and acoustic control. You can prompt the model to shift a neutral recording into a weeping or angry delivery, turn normal dialogue into a whisper, or tweak pitch, speed, and volume. It can even strip regional accents or cleanly add and remove non-verbal sounds like breaths and laughter.

Third is audio cleanup and separation. Feeding a noisy recording with a prompt to remove reverb and background hum will return a dry vocal track. It can also separate singing voices from instrumental music or isolate a specific speaker in a crowded conversation.

Finally, it handles conventional zero-shot voice cloning from just a few seconds of audio, as well as instruction-based speech synthesis generated purely from text descriptions without reference audio.

How to run it

The pipeline pairs a diffusion transformer with the Qwen2.5-Omni-3B multimodal encoder. To make instruction-based tasks easier, Tencent includes a Prompt Enhancer tool that automatically converts rough user prompts into optimized terminal commands and target durations.

Hardware requirements are relatively accessible for a model of this capability. The core weights take up around 6GB, and recent optimizations cut encoder memory usage by roughly 7.5 GiB. With CPU offload enabled, peak VRAM drops from 25 GiB to around 17 GiB, bringing it within reach of standard desktop GPUs.

Ecosystem support arrived on day one. You can run it locally with Gradio or ComfyUI, execute GGUF offline inference via audio.cpp, run it natively on Apple Silicon with MLX, or deploy high-throughput pipelines with vLLM-Omni and SGLang-Omni. Live demos are also hosted on Hugging Face and ModelScope.

Why it matters

For podcasters, video creators, and localization teams, AuK is a huge workflow upgrade. Fixing a misspoken word in post-production no longer requires booking studio time for pickups; you can patch the audio with a single prompt. Anyone dealing with spoken dialogue on a daily basis should give the demo a spin.

Tencent took a smart approach by consolidating generation, denoising, vocal extraction, and surgical editing into one compact 1.5B model instead of chaining multiple fragile tools together. We think the main catch will be precision edge cases, as complex sentences might occasionally yield subtle cadence artifacts, but as an open-source baseline, it sets a formidable standard for conversational audio manipulation.