OpenAI launches MentalHealthBench to grade AI therapists

MentalHealthBench is an open-source evaluation benchmark designed to grade how well AI models handle real-world mental health conversations. Co-developed by OpenAI and over 80 licensed psychologists and psychiatrists from 22 countries, the project gives researchers an objective, expert-crafted framework to measure empathy, safety, and clinical judgment.

While most past evaluations only checked if a chatbot blocked blatant self-harm prompts, MentalHealthBench tests how models navigate the messy spectrum of daily human vulnerability.

What it tests

Using privacy-preserving synthetic conversations, the benchmark covers four distinct user personas: adults, teens aged 13 to 17, caregivers, and clinicians. The teen scenarios were explicitly reviewed by adolescent mental health specialists to ensure responses respect developmental needs.

Scenarios are divided across three acuity tiers. Non-acute tests evaluate handling everyday stress and relationship advice. High-acuity tests simulate severe distress and worsening depression without immediate danger. Emergency tests grade whether the AI correctly identifies crisis signals and routes users to localized real-world hotlines.

How experts grade the models

The 80-plus licensed clinicians involved represent 19 languages and nearly 20 subspecialties. For each synthetic conversation, clinicians created rubrics scoring model behaviors on a scale from -10 to +10.

Positive points reward critical actions like clarifying context and preserving the user's personal agency. Negative points penalize bad medical advice or dismissive tones. Every rubric required agreement from at least two experts without contradiction from a third. In practice, OpenAI uses GPT-5.6 Sol as an automated evaluator to run these expert rubrics against target models.

OpenAI also brought in 44 real-world users who regularly turn to AI for emotional help. Interestingly, users prioritized actionable next steps and tone, while clinicians placed far higher weight on careful context-gathering before offering advice.

Availability

MentalHealthBench is fully open-sourced. AI researchers, developers, and healthcare teams can download the benchmark datasets and rubric structures to audit their own models and identify conversational blind spots.

Why it matters

This benchmark moves the AI safety conversation past generic content filters and into actual conversational nuance. For developers building consumer chatbots or digital support tools, it provides a realistic checklist to keep models supportive without pretending to be doctors.

We think the open release is a smart, transparent move that forces model builders to take emotional guardrails seriously. It will not turn AI into a substitute for a licensed therapist, but it sets a much-needed industry standard for keeping chatbots helpful and harmless when users are at their lowest.