Sword Health researchers created MindEval to test how well large language models handle realistic therapeutic conversations. The framework simulates multi-turn patient-clinician interactions, scores whole conversations against evaluation criteria designed with licensed clinical psychologists, and validates both the patient simulator and the automated judge against human clinicians. The team benchmarked 12 frontier models, naming GPT-5, Claude 4.5, and Gemini 2.5 among them, and reported that average clinical performance stayed below 4 on a 1-6 scale.
The study found performance dropped further in severe symptom scenarios and in longer conversations, noting a decline when conversations extended to 40 turns vs 20. It also reported that larger or reasoning-heavy models did not reliably outperform smaller ones on therapeutic quality. The authors open-sourced prompts, code, scoring logic, and human validation data and said the judge LLM achieved a medium-high average correlation with human evaluators. In discussion threads the lead author, posting as RicardoRei, confirmed prompts were kept the same across models and said the benchmark is intended to measure where models have room for improvement rather than to compare models directly with human clinicians.
Responses from the community highlighted methodological debates and safety concerns. Critics asked for a human clinician control and cautioned that using the same model family to simulate patients and to judge performance can risk epistemic contamination. Others stressed real-world drivers for LLM use in mental health, pointing to availability and cost barriers in traditional care and arguing the need for guardrails. Commenters also noted safety incidents linked to chatbots and urged cautious development. The conversation covered proposals for further validation, including clinician baselines, fine-tuning judge models, and randomized trials versus human therapists, while reiterating that MindEval’s primary contribution is a measurable benchmark for continued research and improvement.
