OpenAI's new AI text detector is only right 26% of the time

Ever since ChatGPT blew up, teachers and editors have been desperately looking for a silver bullet to catch AI-generated text. OpenAI tried to offer one with its own free AI text classifier, pitched as a way to spot homework cheats and automated disinformation campaigns.

It didn't go well. In OpenAI's own benchmark tests on English text, the classifier correctly flagged only 26% of AI-written content, while mistakenly tagging human writing as machine-made 9% of the time. The experiment ended quietly on July 20, 2023, when OpenAI pulled the plug on the tool entirely due to its dismal accuracy.

A detector that couldn't pass its own test

To build the tool, OpenAI fine-tuned a language model on paired datasets of human and AI text covering the same topics. To prevent false accusations, they cranked up the confidence threshold so the system would only speak up when it was extremely certain.

Even so, the guardrails were far too fragile. Anything under 1,000 characters completely broke the system. It fell apart on programming code and performed terribly on non-English languages. If someone fed it predictable content, like a list of prime numbers, it couldn't tell human from machine because the output looked identical.

Worse, users quickly realized that a tiny bit of manual editing could bypass the detector completely. Like many neural networks operating outside their training distribution, the model also had a nasty habit of being extremely confident while making dead-wrong predictions.

OpenAI originally released the classifier as an experimental preview for educators and journalists, pairing it with resources to help teachers navigate ChatGPT in schools. But asking a model to reliably spot its own siblings proved too tall an order.

Why it matters

OpenAI's short-lived detector confirms what researchers have suspected all along: relying on pattern-matching software to police AI writing is a fool's errand. For teachers and content moderators, putting faith in tools with a 9% false-positive rate is a recipe for falsely accusing innocent students and writers.

We think OpenAI made the right call by killing the product rather than giving people a false sense of security. Post-hoc text classifiers simply cannot win against human editing and prompt tweaking, because the attacker only needs to change a few words to break the pattern.

This failure shifts the entire industry's focus away from text classifiers and toward built-in provenance techniques and watermarking, especially for images and audio. If the creators of the leading frontier models can only catch a quarter of their own machine-generated essays, nobody else should pretend they have figured it out either.