Letting AI see where the radiologist looks on a chest X-ray
Overlaying a radiologist’s eye-gaze heatmap on the X-ray improved the accuracy of vision-language models across several chest X-ray tasks.
The question
Computer-aided diagnosis for chest X-rays has become good, but the conversation between the AI and the radiologist is usually one-way, with the radiologist only supplying the image. What if the AI also knew where the radiologist was looking?
What we did
We recorded eye-gaze data from radiologists and turned it into heatmaps that we overlaid on the X-ray, highlighting the areas of intense focus. We gave these, together with text prompts, to vision-language models and tested four tasks: visual question answering, report automation, error detection and differential diagnosis.
What we found
Adding eye gaze significantly improved the accuracy of the analysis. Fine-tuning with gaze also outperformed other medical vision-language models on every task except visual question answering.
Why it matters
A radiologist’s attention is expert knowledge that the AI can learn from. This points to a more human-centred way of building diagnostic AI. It is the starting point for the later Look & Mark work.