Google's new music model turns your Mac into a live AI instrument

Google just open-sourced Magenta RealTime 2 (MRT2), an interactive generative music model designed to be played like an actual instrument rather than treated like an offline prompt-to-song renderer.

The release includes two model sizes (2.4B and 230M parameters) capable of continuous streaming audio generation with roughly 200ms latency. You can guide it with text prompts or reference audio, but the real magic is real-time MIDI input: you press a key on your keyboard, and the model synthesizes sound on the fly. Note that it only outputs instrumental music; vocal synthesis is not supported.

What it does and how it works

Most generative audio models are built like batch jobs: you submit a prompt, wait for tokens to generate, and download an audio file. MRT2 is built from the ground up for low-latency live interaction.

Under the hood, MRT2 operates as an autoregressive codec language model using the SpectroStream codec. To keep memory bounded during continuous performance, it uses a causal sliding window attention mechanism alongside learnable attention embeddings that prevent long-context feedback artifacts.

The real breakthrough is frame-aligned conditioning. Style prompts are embedded via MusicCoCa, while incoming MIDI notes are injected directly at every single 40ms generation step. This allows the model to react to your playing almost instantly, achieving around 15x lower latency compared to the first-generation Magenta RealTime.

Hardware specs and how to run it

While the original Magenta model required cloud TPUs or heavy server GPUs, MRT2 runs entirely locally on everyday musician hardware: Apple Silicon Macs.

Google built a dedicated C++ inference engine on top of Apple’s MLX framework. The model is compiled into an MLX container that executes directly on the Mac's unified GPU. It handles state caching, audio buffering, resampling, and MIDI routing natively without roundtrips to Python.

Google has released the open weights, a Python library (pip install magenta-rt) supporting JAX and MLX, the C++ streaming engine, and standalone demo apps. Because the engine is C++, it can also be embedded directly into digital audio workstations (DAWs) or custom plugin wrappers.

To test it out, Mac users with M-series chips can download the prebuilt standalone apps or install the Python package to experiment with sound cloning and live accompaniment.

Why it matters

We think Google made a brilliant strategic pivot here: instead of racing Suno or Udio to build an AI jukebox that spits out complete songs, MRT2 treats generative AI as a dynamic synth module inside a musician's existing rig. Real musicians don't want a robot to write their whole track; they want expressive, unpredictable new sounds they can play with their hands.

Running fully local on Apple Silicon solves the latency and subscription friction in one fell swoop. A 200ms latency is still a bit sluggish for tight live keyboard solos, and the lack of vocal synthesis is a clear limitation. But as an open-weights foundation for a brand new category of playable AI instruments, MRT2 is arguably the most musician-friendly release we have seen this year.