Building realistic simulated patients to evaluate mental health AI (DIAL)

A 2025 preprint introduces DIAL, a method for realistic multi-turn dialogue simulation, addressing the unrealistic patient behavior produced by prompting frontier models.

DIAL, an adversarial training framework, produces simulated users whose failure rates closely track those of real users — enabling more reliable, lower-cost evaluation of mental health AI before deployment.

Study type

Methodological (machine learning)

Published in

arXiv

Partner

University of Cambridge, New York University

Citation

Zhu, Z., Tieleman, O., Stamatis, C. A., Smyth, L., Hull, T. D., Cahn, D. R., Chen, J., & Malgaroli, M. (2025). DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation [Preprint]. arXiv. https://arxiv.org/abs/2512.20773

A 2025 arXiv preprint introduces Direct Iterative Adversarial Learning (DIAL), a framework that makes simulated users more realistic by pitting a user-simulator generator against a discriminator. Applied to mental health support — where realistic user behavior is essential for surfacing failures — DIAL restored lexical diversity lost in supervised fine-tuning and produced simulated failure rates that closely matched real-world rates, supporting more reliable and cost-effective system evaluation before deployment.

Because the simulator's failure modes mirror real usage with low distributional divergence, teams can stress-test mental health AI more faithfully before release, rather than relying on simulation sets whose relationship to real conversations is unknown.