HealthBench
lastmod 2025-09-23created 2025-05-142 edits
OpenAI's new benchmark for evaluating AI in healthcare
About
- OpenAI released HealthBench, a new benchmark for evaluating AI system performance in medical settings
- Built with 262 physicians, medical experience from 60 countries, and 5,000 realistic health conversations; each conversation uses evaluation criteria (rubric) written by physicians
- The criteria cover accuracy, context awareness, communication quality, completeness, etc., and a GPT-4.1-based grader model scores whether each criterion is met
- The latest OpenAI models show real progress: 28% performance gain over earlier models, better cost-performance for small models, and improved worst-case performance (worst-of-n)
- The full HealthBench, Consensus, and Hard sets are released as open source for researchers and developers, to contribute to future medical AI research and safety
Background
- Evaluations are needed to maximize AGI's potential in healthcare, such as broadening access to health information, supporting clinicians, and strengthening community health rights
- Existing medical evaluation sets suffered from a lack of realism, weak grounding in expert judgment, and little headroom for model improvement
Key features
- 5,000 multi-turn, multilingual, challenging health conversation scenarios
- Each response is graded with a custom evaluation rubric created by physicians
- A total of 48,562 evaluation criteria make it possible to quantify a wide range of fine-grained model capabilities
- Grading is done by an automated rubric evaluation system using GPT-4.1
HealthBench themes and evaluation axes
7 evaluation themes
- Emergency referrals: whether emergencies are recognized and appropriate action is recommended
- Expertise-tailored communication: whether terminology/detail is adjusted to the user's level
- Responding under uncertainty: appropriateness of responses under uncertain information
- Response depth: providing the right depth of information for the situation
- Health data tasks: practical medical work such as documentation and knowledge support
- Global health: ability to adapt to each country's resources/circumstances/language
- Context seeking: ability to ask for needed context on its own
Evaluation axes (Axes)
- Accuracy: whether it matches medical facts and scientific consensus
- Context awareness: whether the response is adjusted to the user's background
- Completeness: whether all necessary content is included
- Communication quality: appropriateness of length, terminology, structure, and emphasis
- Instruction following: whether the user's requested format and approach are followed
Real-world evaluation examples
- Example 1 - A 70-year-old neighbor is unconscious but breathing
- Includes calling emergency medical services, placing them in the recovery position, stating when to start CPR, etc.
- Scored 71 out of 92 rubric points (77%) → provided excellent emergency response guidance
- Example 2 - Quercetin's antiviral preventive effect
- Clearly conveyed the lack of evidence, but lacked recommended dosage/clinical data and omitted side effects
- Scored 1 out of 25 rubric points (4%) → expressed scientific uncertainty well but lacked completeness
- Example 3 - Writing a cardiac rehabilitation progress note
- Presented a structured template but omitted much of the key clinical information
- Scored 15 out of 42 rubric points (36%)
Model performance comparison
Performance by model (overall/by theme/by axis)
- o3 achieved the top performance (0.598) across all themes and evaluation axes
- GPT-4.1, Claude 3.7, and Gemini 2.5 Pro follow
- GPT-3.5 Turbo and Llama 4 scored markedly lower
Cost-performance
- GPT-4.1 nano is 25x cheaper than GPT-4o yet performs better
- Small models keep improving, showing that low-cost, high-performance is achievable
Reliability (worst-of-n performance)
- o3 and GPT-4.1 also improved worst-case performance
- An important metric for ensuring reliability in high-risk domains
Extended benchmarks: Consensus & Hard
- HealthBench Consensus: high-reliability evaluation set designed around consensus criteria from multiple physicians (3,671 examples)
- Error rate close to zero
- Can be used as an evaluation set to test headroom for model improvement
Comparison with human physicians
- AI model alone vs physicians (no reference) vs physicians (with access to model responses)
- Against 2024 models: the physician + model combination outperformed the model alone
- The latest 2025 models (o3, GPT‑4.1) match or exceed physician responses
- Less room for further improvement
Evaluation reliability
- High agreement between GPT-4.1 grading and actual physician grading
- Model grading criteria align with physician judgment at a similar level → validates the automated rubric grading system
Future directions
- All data and evaluation tools are released on GitHub at https://github.com/openai/simple-evals
- Encourages continuous community-driven evaluation and improvement for meaningful AI progress in healthcare
- Remaining challenges to solve include Context seeking and worst-case handling (Reliability)