Are language-technology studies on radiology reports described well enough to repeat?
Of 164 studies, few shared data or code or tested their work externally, so we called for reporting standards specific to clinical NLP.
The question
If a study does not describe its data, methods and checks clearly, others cannot compare it or reproduce it. We asked how well studies applying language technology to radiology reports describe themselves.
What we did
A team including NLP researchers and an epidemiologist searched for studies published between 2015 and October 2019, screened thousands of records, and assessed the included studies against 15 criteria covering data source, datasets, ground truth, outcomes and reproducibility.
What we found
We included 164 studies from 4,836 records. The commonest applications were disease information or classification (28%) and diagnostic surveillance (27.4%), and 86% used English reports. Few reported external validation (10.8%), data availability (8.5%) or code availability (9.1%), and only about half reported precision (48.8%) and recall (53.7%).
Why it matters
Reporting quality fell short, which prevents comparison, reproducibility and replication. We argued for reporting standards specific to clinical NLP studies. This complements the companion review of what the field has built.