Elon's Grok caught quoting OpenAI's policy

Imagine picking a fight with your biggest rival, only for your new product to start quoting their rulebook word-for-word.

That exact nightmare played out for Elon Musk’s xAI this past week. Just days after Grok went into wide release, security tester Jax Winterbourne shared a bizarre screenshot where Grok refused a prompt by stating, “I’m afraid I cannot fulfill that request, as it goes against OpenAI’s use case policy.”

For a bot explicitly built to take down ChatGPT, channeling OpenAI’s ghost was the last thing anyone expected.

Web scrape accident or dirty fine-tuning?

xAI didn’t deny the screenshot. Company engineer Igor Babuschkin jumped into the replies to smooth things over, claiming that because the internet is flooded with ChatGPT outputs, xAI accidentally scraped some of them while pre-training Grok on vast amounts of web data.

Babuschkin insisted the team was surprised when they first saw it, claimed the glitch was very rare, promised to fix it in upcoming updates, and swore that no OpenAI code went into Grok.

Winterbourne did not buy the rarity excuse, pointing out that this canned refusal pops up quite frequently whenever code generation is involved.

AI researcher Simon Willison told Ars Technica that Babuschkin’s web scraping defense seems highly unlikely. Large language models do not typically regurgitate stray training text verbatim unless they have been explicitly instruction-tuned on it. Willison noted that dozens of open-weights models on Hugging Face behave like ChatGPT for one simple reason: they were fine-tuned on datasets generated by OpenAI APIs or scraped directly from ChatGPT.

Why it matters

We think this slip-up exposes one of the AI race’s dirtiest open secrets: nearly everyone is quietly distilling OpenAI’s data while claiming total independence.

For users and developers paying to test Grok, this is a clear sign to temper expectations. If you were hoping for a genuinely distinct intelligence with its own raw reasoning, Grok’s guardrails and coding logic still look suspiciously like a knockoff ChatGPT under the hood.

Industry-wide, this is wildly embarrassing for Musk’s anti-woke, anti-OpenAI branding. Building an AI challenger is hard enough, but catching your rebel bot dutifully reciting your arch-rival’s terms of service is pure comedy.

More broadly, it highlights how contaminated the training web has already become. If even well-funded labs cannot separate synthetic ChatGPT outputs from human data, downstream models risk degenerating into copies of copies. Cleaning training data is about to become the single most expensive bottleneck in frontier AI.