Tencent Hunyuan open-sources real-time interpreter that clones your voice

Hy-SimulMT is an open-source model from Tencent Hunyuan designed for real-time, bidirectional Chinese-English simultaneous translation. It streams translations in both text and synthesized speech as someone speaks, and it clones the original speaker's vocal tone into the target language.

What it does

Standard live translation tools usually chain together speech-to-text, machine translation, and text-to-speech. That setup typically forces the system to wait for a full sentence before translating, introducing awkward silences.

Hy-SimulMT handles incoming audio as a continuous stream. Instead of waiting for pauses, it begins translating incrementally. To keep latency minimal, it starts speaking early and makes slight inline self-corrections as more context arrives, mirroring how human simultaneous interpreters operate.

When paired with a TTS service, it also performs real-time voice cloning. In a multi-speaker conference call, listeners can immediately recognize who is talking because the translated voice retains the original speaker's distinctive pitch and cadence.

Benchmarks and specs

The model is built on a 30B Mixture-of-Experts architecture that activates only about 3B parameters per token. Weights are available in BF16, FP8, and GGUF formats. While the uncompressed model takes up about 61GB of disk space, the Q4 GGUF build drops to roughly 20GB.

On the open-source RealSI long-form benchmark, Hy-SimulMT recorded a StreamLAAL latency of roughly 3 seconds, lower than the other compared open-source models and commercial APIs. Its GEMBA translation quality score beat all compared open-source baselines and four out of five commercial services. It also led the pack on the RealSI-Sentence benchmark with an 84.7 score.

For real-world validation, Tencent evaluated the system on 18 audio recordings across ten domains, including AI, medicine, and manufacturing. When tested alongside human interpreters holding CATTI Level 1 and Level 2 certificates, the model delivered comparable translation fidelity while responding with lower latency.

How to run it

The code is licensed under Apache 2.0, with weights hosted on Hugging Face and ModelScope. The repository includes the SimulInfer inference engine and SimulTrain fine-tuning scripts.

Running it locally requires a Linux setup with Python 3.12+, an NVIDIA GPU with enough VRAM, and vLLM version 0.31 or higher. Running deploy.sh inside the SimulInfer directory configures dependencies and spins up a WebSocket-ready web interface. For a zero-setup trial, Tencent also offers the model inside its mobile Hy Translate App.

Why it matters

If you build tools for cross-border webinars, global meetings, or live streaming, Hy-SimulMT provides a ready-made pipeline that eliminates the clumsy delays of multi-step transcription setups. The MoE design keeps memory demands modest, meaning the 20GB quantized model can feasibly run on single-GPU developer hardware.

It makes live cross-language communication feel far more conversational by preserving the speaker's voice. Still, remember that full audio generation depends on an external TTS service, and automated interpretation remains best suited for low-risk collaboration rather than mission-critical legal or medical negotiations.