This is the implementation of Generating Radiology Reports via Memory-driven Transformer at EMNLP-2020.

• The codebase is kind of old so we refer the readers to this awesome project (ViLMedic). You can also check our newly released PTUnifier, which can perform various medical image-text tasks like radiology report generation.
• The codes for visualization and clinical efficacy are updated.

If you use or extend our work, please cite our paper at EMNLP-2020.

@inproceedings{chen-emnlp-2020-r2gen,
    title = "Generating Radiology Reports via Memory-driven Transformer",
    author = "Chen, Zhihong and
      Song, Yan  and
      Chang, Tsung-Hui and
      Wan, Xiang",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2020",
}

• torch==1.7.1
• torchvision==0.8.2
• opencv-python==4.4.0.42

You can download the models we trained for each dataset from here.

We use two datasets (IU X-Ray and MIMIC-CXR) in our paper.

For IU X-Ray, you can download the dataset from here and then put the files in data/iu_xray.

For MIMIC-CXR, you can download the dataset from here and then put the files in data/mimic_cxr. You can apply the dataset here with your license of PhysioNet.

NOTE: The IU X-Ray dataset is of small size, and thus the variance of the results is large. There have been some works using MIMIC-CXR only and treating the whole IU X-Ray dataset as an extra test set.

Run bash train_iu_xray.sh to train a model on the IU X-Ray data.

Run bash train_mimic_cxr.sh to train a model on the MIMIC-CXR data.

Run bash test_iu_xray.sh to test a model on the IU X-Ray data.

Run bash test_mimic_cxr.sh to test a model on the MIMIC-CXR data.

Follow CheXpert or CheXbert to extract the labels and then run python compute_ce.py. Note that there are several steps that might accumulate the errors for the computation, e.g., the labelling error and the label conversion. We refer the readers to those new metrics, e.g., RadGraph and RadCliQ.

Run bash plot_mimic_cxr.sh to visualize the attention maps on the MIMIC-CXR data.