OpenAI uses GPT-4 to catch GPT-4's mistakes

As AI models get smarter, their mistakes get subtler. When a chatbot writes code that looks completely fine to a human reviewer but secretly contains a nasty security flaw, our standard alignment playbooks start to break down.

OpenAI just revealed CriticGPT, a new model trained specifically to find bugs in ChatGPT's code outputs.

Training an AI on poisoned code

Modern chatbots rely heavily on Reinforcement Learning from Human Feedback, where human trainers grade model responses against each other. But GPT-4 has gotten good enough that human annotators often miss subtle bugs in its answers.

To build a better pair of eyes, OpenAI had trainers deliberately inject bugs into ChatGPT's code, then trained CriticGPT to spot them.

In testing, human trainers assisted by CriticGPT outperformed unassisted reviewers 60% of the time. When reviewing naturally occurring ChatGPT bugs, human trainers preferred CriticGPT's critiques over standard ChatGPT self-critiques 63% of the time, noting it was less prone to nitpicking and hallucinations.

By tweaking test-time search against a critique reward model, OpenAI can also dial between aggressively hunting down rare bugs or playing it safe to avoid false alarms.

OpenAI is now integrating CriticGPT-style systems into its core labeling pipeline, though they readily admit limitations. CriticGPT was trained on short snippets, struggles with dispersed errors that span an entire codebase, and sometimes hallucinates bugs that confuse human reviewers.

Why it matters

We think this is one of the clearest signs yet that pure human supervision has hit a ceiling. When models write code at or above the median human level, asking a tired contractor to spot edge cases on a timer is a recipe for silent failure.

For developers, an AI trained specifically as a rigorous code reviewer is arguably more valuable than another autocomplete tool. What teams need is not more syntax suggestions, but a ruthless auditor that catches logic traps before pull requests merge.

However, supervising models with other models carries an obvious risk. If CriticGPT hallucinates a plausible-sounding flaw, it can gaslight human annotators into agreeing with it. The moment trainers start blindly trusting the critic model to save time, the entire alignment pipeline risks drifting into a blind spot where only the AI knows what is actually broken.