A realistic test for AI that diagnoses rare diseases

The question

Medical AI is promising at reading clinical text and supporting diagnosis. But it is often tested on small or artificial sets of questions that do not look like the messy cases doctors really face. For rare diseases, where even specialists struggle, that is a particular problem.

What we did

We gathered thousands of real clinical case reports from PubMed Central and turned them into tasks that mimic real clinical problem-solving. These include screening a case for possible rare diseases and confirming a specific diagnosis from a free-text description. The result is a benchmark called RareArena.

What we found

RareArena lets different AI models be compared on the same realistic cases, showing where they are strong and where they fall short at understanding medical language and recognising rare conditions.

Why it matters

A trustworthy test comes before a trustworthy tool. A realistic testing ground helps developers build safer AI that could help clinicians recognise rare diseases earlier and more accurately.