Mirelo and Kyutai just turned full songs into multitrack MIDI

MuScriptor is a new open-source audio-to-MIDI model built jointly by Mirelo and Kyutai. Put simply, it takes a fully mixed audio recording and turns it into clean, separated MIDI tracks for each instrument, including vocals, drums, bass, and keys.

Most existing audio-to-MIDI tools stumble when music gets complex. They do fine on an isolated vocal stem or a solo piano recording, but completely fall apart once you throw in a dense song where bass, guitars, and percussions are layered together under heavy reverb and compression. MuScriptor bypasses the need to split stems beforehand and works directly on the full stereo mix.

What it does

You feed it an MP3 or WAV file of a finished song, and it spits out discrete MIDI data for the separate instruments playing across the mix. You can export individual MIDI clips straight into your DAW to remix, swap synth patches, or build entirely new tracks. It also generates full sheet music if you just want to learn how to play a specific part.

Beyond basic pitch and note length, the system detects chords, musical key, and tempo, handing producers the foundational musical context alongside raw notes.

Under the hood

MuScriptor approaches music transcription as a standard language modeling problem using a straightforward decoder-only transformer backbone.

It takes a 5-second audio chunk represented as a mel-spectrogram for prefix conditioning, then autoregressively spits out discrete tokens that capture note pitch, onset, offset, and instrument type—basically a 1D tokenized piano roll. To solve the chronic shortage of accurately labeled multi-instrument audio, the team devised a three-stage training recipe: synthetic pre-training (DSynth), fine-tuning on real music audio (DReal), and reinforcement learning post-training (DRL).

According to Mirelo, pre-training on synthetic data alone matched the previous state-of-the-art multi-instrument baseline YourMT3+. Fine-tuning on real music then pushed all core metrics up by roughly 20 percentage points, while the RL post-training dialed in maximum precision. The model also features an optional instrument conditioning hook to steer outputs based on specific instrument setups.

How to get it

The inference code is hosted on GitHub, model weights are up on Hugging Face, and technical documentation is available via their arXiv paper.

If you prefer not to spin up local Python environments, Mirelo also trained an upgraded, higher-accuracy checkpoint on even more data. That production model is accessible for free through their Mirelo Studio web app.

Why it matters

For bedroom producers, beatmakers, and learners, transcribing complex arrangements has always been a painful, ear-straining chore. Having a system spit out usable MIDI clips across a whole arrangement saves hours of tedious note-plotting and stem-ripping.

We think pairing a decoder-only LLM setup with reinforcement learning post-training is a genuinely smart design for audio transcription. It avoids convoluted multi-stage signal processing pipelines and leans into pure data scaling. While heavily distorted guitars or ultra-dense hyperpop layers might still challenge note precision, this is easily one of the most practical and capable tools music makers should test immediately.