A clever prompt trick gets ChatGPT to sing Beatles duet

OpenAI gave its new Advanced Voice Mode a very clear rule before launch: do not sing under any circumstances. In fact, leaked system prompts show the explicit instruction was literally "Do not sing or hum."

It took users less than 48 hours to break that rule.

On Thursday, AJ Smith, an associate director of AI engineering at S&P Global, posted a video of himself playing guitar while ChatGPT casually joined him for a duet of The Beatles’ 1966 classic “Eleanor Rigby.”

The four-chord trick that broke the guardrails

If you ask Advanced Voice Mode to sing directly, it will shut you down immediately with a canned response about guidelines.

Smith took a clever detour instead. He told the bot they were going to play a game: he would play the classic four pop chords on his guitar, and the AI would suggest song titles for him to sing along to.

After cycling through a few tracks, the AI unexpectedly joined in on its own. It belted out the opening line, “Ah, look at all the lonely people,” while hyping up Smith's guitar playing between phrases.

Smith admitted the moment gave him literal chills, especially because he never explicitly instructed it to harmonize. The pitch was slightly shaky and later dissolved into rhythmic reciting, but the melodic intent was unmistakable. Ars Technica reporters later managed to coax the model into a pitch-perfect rendition of “Happy Birthday” as well.

Why OpenAI put a muzzle on melodies

OpenAI did not muzzle AVM's singing voice because of technical limitations. It did so out of legal fear.

Because AVM is an end-to-end multimodal model trained directly on audio tokens, it has absorbed vast amounts of copyrighted audio, including commercial recordings. Letting users freely prompt the model to belt out protected songs is an express ticket to massive copyright lawsuits from major record labels.

What Smith pulled off is essentially a classic prompt injection, cloaked in social engineering. By framing the prompt as an interactive chord game, the model's contextual momentum completely overpowered its rigid negative constraints.

Why it matters

This Beatles duet is an incredible glimpse into what true end-to-end audio intelligence feels like. Unlike old text-to-speech pipelines, AVM actually feels the rhythm, recognizes chord progression, and improvises vocal timing on the fly.

For musicians and creatives, it shows that AI jamming partners are already technically feasible, even if you currently have to trick the software into doing it. Don't expect this to become a reliable studio workflow just yet, as OpenAI will almost certainly patch this specific loophole.

For the broader industry, it exposes the fundamental fragility of model guardrails. Trying to stop an audio-native model from singing by telling it "do not sing" in a text prompt is like telling a toddler not to look at a wrapped present.

We think OpenAI is in a tough spot here. Restricting raw model capabilities through superficial behavioral rules hurts the user experience without actually solving the copyright headache. The music industry is notoriously litigious, and every time an AI accidentally remembers a Beatles hook, the target on OpenAI's back gets a little bigger.