OpenAI's guide to fixing unreliable enterprise AI with evals
OpenAI just published a playbook for business leaders on turning messy AI pilots into reliable production workflows using contextual evals. The framework details how cross-functional teams should build golden test sets, blend LLM graders with human review, and spin up a continuous data flywheel. It basically argues that as foundation models commoditize, proprietary evaluation criteria will become the real enterprise moat.
Most companies deploying AI hit the same wall: the prototype looks incredible in a demo, but the moment it touches real customers or internal workflows, it starts hallucinating or missing the mark.
Over one million businesses are now using AI, but OpenAI admits that many are struggling to see real business results. On November 19, the company published a framework breaking down how they tackle this problem internally, and the answer comes down to one core concept: contextual evaluations (evals).
OpenAI points out that frontier benchmarks—the tests used to rank raw model intelligence—cannot tell you if a model will handle your company’s specific sales emails or onboarding pipeline. To get measurable ROI, businesses have to build their own custom evaluation rubrics.
Skip the perfectionism and build a golden set
OpenAI’s recipe starts with a cross-functional squad. Leaving AI deployment strictly to engineers is a common trap, because technical leads rarely know what a good sales conversation sounds like or which compliance guardrails actually matter.
Instead, domain experts and engineers should define the objective in plain terms. Then comes the messy part: reviewing 50 to 100 early outputs to run an error analysis and categorize failure modes.
From there, teams should create a "golden set" of ideal inputs and outputs, representing the judgment and taste of their top performers. While companies can scale grading by having an LLM judge system outputs, OpenAI emphasizes that human experts must regularly audit the LLM grader and manually review edge cases.
The data flywheel becomes your real moat
Deployment isn’t the finish line; it is where the evaluation flywheel actually begins. OpenAI advises teams to continuously log live production inputs, outputs, and outcomes.
When the system flags edge cases or high-stakes interactions, they should route automatically to domain experts for review. Those expert judgments then get fed back into the eval set, the prompt architecture, and the tooling.
Over time, this loop generates a proprietary, context-heavy dataset tailored specifically to that company’s business operations—something a competitor simply cannot copy by calling the same raw foundation API.
Why it matters
Most teams spend weeks tweaking prompts and swapping base models, but spend almost no time defining what success actually looks like in numbers. Without systematic evals, companies stay trapped between shipping risky tools that burn customer trust or keeping AI stuck in eternal sandbox experiments.
OpenAI is essentially telling business leaders that traditional management rigor is the most critical AI skill. If you cannot explicitly describe what "great" execution looks like in your workflow, a smarter model will not solve your problem.
That said, executing this framework is demanding. Building and maintaining a living eval pipeline requires significant domain expert hours, which can quickly strain leaner teams. For smaller startups, setting up a heavyweight data flywheel before proving product-market fit could easily become an expensive distraction.
Sources (1)
- openai.com(opens in a new tab)Originalopenai.com