1. Introduction
The impact of the COVID-19 pandemic on the health and economy has been unprecedented. While more than one year has been passed since the declaration of the global pandemic by the World Health Organization [
1], countries are still struggling with controlling the spread of the SARS-CoV-2 virus that is causing the pandemic. In this regard, the global healthcare system has suffered a devastating impact from this pandemic, with hospitals and clinics overwhelmed by the surge of patients, and thus not all patients can have access to intensive care units for further treatment and care [
2]. Furthermore, there have been shortages in PPEs, ventilators, and other medical supplies due to the increasing demand on healthcare resources [
2,
3].
This significant strain on healthcare resources caused by the COVID-19 pandemic from both a personnel and supplies perspective necessitates improved clinical decision support tools for aiding clinicians and front-line healthcare workers with more efficient and effective clinical resource allocation. A critical part of clinical resource allocation during this pandemic has been severity assessment, which is often conducted with the assistance of chest X-ray (CXR) images. More specifically, radiology indicators within the patient’s lungs such as ground-glass opacities can provide critical information for determining whether the condition of a SARS-CoV-2 positive patients warrants advanced care such as ICU admission and ventilator administration.
Given that severity assessment using CXR images can be quite challenging even for expert radiologists, providing computer-aided clinical support for this task can greatly benefit the hospitals in determining patients’ conditions and respond more quickly to those who may require more advanced treatments or intensive care. While much of research literature have focused on computer-aided COVID-19 detection from CXR images [
4,
5] and computed tomography (CT) scans [
6,
7,
8,
9,
10], the area of computer-aided severity assessment is significantly less well explored. Some of the notable works in this area are COVID-Net S, a tailored deep convolutional neural network proposed by Wong et al. [
11] to predict the extent scores from CXR images, as well as the study by Cohen et al. [
12] where the same extent scores are predicted.
Motivated to extend upon this area in the direction of airspace disease grading, we introduce COVID-Net CXR-S, a convolutional neural network for predicting the airspace severity of a SARS-CoV-2 positive patient based on a CXR image of the patient’s chest, as part of the COVID-Net open source initiative [
4,
7,
8,
11]. The paper is organized as follows.
Section 2 describes the methodology behind the design and construction of the proposed COVID-Net CXR-S, a demographic and protocol analysis of the dataset used, as well as radiologist validation.
Section 3 presents and discusses the quantitative and qualitative results obtained from the experiments evaluating the efficacy of the proposed COVID-Net CXR-S.
2. Material and Methods
In this work, we introduce COVID-Net CXR-S, a convolutional neural network tailored for the prediction of airspace severity of a SARS-CoV-2 positive patient based on chest X-ray images. To train COVID-Net CXR-S, we transferred representational knowledge from CXR images of a multinational cohort, and then leveraged the CXR data grouped based on airspace severity levels. We further validated the behavior of COVID-Net CXR-S in a transparent and responsible manner via explainability-driven performance validation, as well as conducted radiologist validation on select cases by two expert board-certified radiologists. The details between network design, data preparation, explainability-driven performance validation, and radiologist validation are described below.
2.1. Network Design
The proposed COVID-Net CXR-S architecture is depicted in
Figure 1, with additional details on each layer within the architecture detailed in
Table 1. More specifically, we leveraged a machine-driven design exploration strategy [
13] to construct a backbone architecture tailored for a strong balance between accuracy and efficiency [
14]. The constructed backbone architecture exhibits light-weight macro-architecture and micro-architecture designs comprised primarily of depthwise and pointwise convolutions with selective long-range connectivity. We then leveraged transfer learning to transfer representational knowledge gained from over 16,000 CXR images from a multinational cohort of over 15,000 SARS-CoV-2 positive and negative patient cases [
15,
16,
17,
18,
19,
20] via the constructed backbone architecture into a custom network architecture for severity assessment, where a combination of a dense layer and a severity prediction layer is used to predict between two levels of airspace severity. Specifically, 16,352 CXR images with 2358 SARS-CoV-2 positive images and 1505 positive patient cases were leveraged in the transfer learning. The multi-national patient cohort curated by the Radiological Society of North America (RSNA) RICORD initiative [
20] was used afterwards for the evaluation of the efficacy of the proposed network.
2.2. Data Preparation
In this study, we leveraged the RSNA RICORD (International COVID-19 Open Radiology Database) dataset [
20] for the severity scoring to train COVID-Net CXR-S after transfer learning. The RSNA RICORD dataset was curated by the Radiological Society of North America (RSNA) as part of a global initiative to assemble an international task force of scientists and radiologists to create a multi-institutional, multinational, expert-annotated COVID-19 imaging dataset. More specifically, we leveraged the airspace disease grading provided by RSNA RICORD, where each lung is split into three separate zones (for a total of six zones) and opacity is studied for each zone. We group the patient cases into two airspace severity level groups: (1) Level 1: opacities in 1–2 lung zones, and (2) Level 2: opacities in 3 or more lung zones. Example CXR images for the different airspace severity level groups from the dataset are shown in
Figure 2. This severity level designation was chosen given clinical similarities between patient cases within each airspace severity level groups in terms of the treatment regimen, and thus facilitates clearer guidelines for course of action.
Given this airspace severity level grouping scheme, the RSNA RICORD dataset used in this study consists of 909 CXR images from 258 patients, with 227 images from 129 patients in Level 1 and 682 images from 184 patients in Level 2. We used 150 randomly selected CXR images for the test set, ensuring no patient overlap between the test and train datasets.
Table 2 summarizes the demographic variables and imaging protocol variables of the CXR data in the RSNA RICORD dataset used in this study. It can be observed that the patient cases in the cohort used in the study are distributed across the different age groups, with the mean age being 59.11 and the highest number of patients in the cohort are between the ages of 50–69.
2.3. Network Training
Training on the COVID-Net CXR-S architecture after knowledge transfer is conducted on the training portion of the aforementioned dataset using the Adam optimizer with a learning rate of 0.0001 for 30 epochs with a batch size of 32. During the training, we account for the imbalance in the number of patient cases between the airspace severity levels by performing batch balancing, where at each epoch we randomly sampled an equal number of CXR images from each severity level for each batch of data. As a pre-processing step, the CXR images were cropped (top 8% of the image) prior to the training process to better mitigate the influence of commonly-found embedded textual information, and resampled to 480 × 480 for training purposes. In addition, we leveraged data augmentation during the training process with the following augmentation types: translation (±10% in x and y directions), rotation (±), horizontal flip, zoom (±15%), and intensity shift (±10%).
2.4. Explainability-Driven Performance Validation
To study the decision-making behavior of the proposed COVID-Net CXR-S network in a transparent and responsible manner, we conducted explainability-driven performance validation using GSInquire [
21], which was shown to provide state-of-the-art explanations. More specifically, GSInquire leverages the concept of generative synthesis [
13] from the machine-driven design exploration process via an inquisitor
within a generator-inquisitor pair
to generate quantitative interpretations of the decision-making process of COVID-Net CXR-S on a given CXR image. In this case, the generator
in the generator-inquisitor pair is the optimal generator that was leveraged to construct the backbone architecture during the network design process. The details pertaining to GSInquire for explaining the decision-making behavior of deep neural networks on CXR images can be found in [
4]. An interesting property of GSInquire that also makes it well-suited for explainability-driven performance validation is that it is capable of producing explanations identifying specific critical factors within an image that quantitatively impacts the decisions made by a deep neural network, thus making it more readily interpretable and more quantitative for validation purposes than the types of relative importance variations visualized by other methods.
This explainability-driven performance validation process enables the identification of anomalies in decision-making behavior or potential erroneous indicators, leading to invalidate decisions or biases, as well as the validation of whether clinically relevant indicators are leveraged.
2.5. Radiologist Validation
The results obtained for COVID-Net CXR-S during the explainability-driven performance validation process for selected patient cases are further reviewed and reported on by two board-certified radiologists (A.S. and A.A.). The first radiologist (A.S.) has over 10 years of experience, and the second radiologist (A.A.) has over 19 years of radiology experience.
3. Results and Discussion
To evaluate the efficacy of the proposed COVID-Net CXR-S for the purpose of severity assessment of COVID-19 cases from CXR images, we conducted both quantitative performance evaluation as well as qualitative explainability-driven performance validation. The quantitative and qualitative results are presented and discussed below.
3.1. Quantitative Results
The overall quantitative performance assessment results of the proposed COVID-Net CXR-S network on the RSNA RICORD dataset can be seen in
Table 3. For comparison purposes, quantitative performance assessment was also conducted on the ResNet-50 [
22] network architecture as well as on CheXNet [
23], a state-of-the-art deep neural network architecture that has been shown to outperform other network architectures for CXR image analysis tasks. The architectural and computational complexity of COVID-Net CXR-S in comparison to CheXNet [
23] and ResNet-50 [
22] is also shown in
Table 4.
A number of observations can be made from the quantitative results. More specifically, it can be observed that the COVID-Net CXR-S network can achieve a high level of accuracy at 92.66%, which is 9.33% and 5.33% higher than that achieved by CheXNet and ResNet-50 network architectures, respectively. The COVID-Net CXR-S network also achieved strong performance in terms of area under the receiver operating characteristic curve (AUC ROC), achieving a 96.09% AUC which is 12.47% and 4.21% higher than the CheXNet and ResNet-50 networks, respectively. The COVID-Net CXR-S network achieved higher accuracy and AUC at a significantly lower computational complexity of ∼57% and ∼69% (at ∼11.1 G FLOPs) in comparison to the CheXNet and ResNet-50 networks, respectively, as well as a ∼63% lower architectural complexity (at ∼8.8M parameters) than the ResNet-50 network, and achieving similar architectural complexity when compared to CheXNet. Achieving high architectural efficiency and computational efficiency is important for operation in resource-limited clinical scenarios where low-cost computing devices are desired.
It can also be observed that the COVID-Net CXR-S network achieved a sensitivity of
on the Level 2 cases and sensitivity of
on the Level 1 cases. Furthermore, it can also be seen that the proposed COVID-Net CXR-S network can achieve high positive predictive value (PPV) of 95.78% and 87.27% for Level 2 cases and Level 1 cases, respectively. This high PPV for Level 2 cases ensures that fewer false positives for more severe cases are reported by COVID-Net CXR-S, which is important since patients with severe conditions require advanced treatment and management and thus high false positive rate can lead to significant burden on the healthcare system where resources are limited. Finally,
Table 5 provides a more detailed picture of the performance of COVID-Net CXR-S via the confusion matrix.
Therefore, it can be clearly seen that the COVID-Net CXR-S can successfully predict the severity of COVID-19 infections and distinguish COVID-19 infections from CXR images to make such predictions as evident by the predictive performance shown in
Table 3 and the critical factors it leveraged as identified by
Figure 3.
3.2. Qualitative Results
Results from the conducted explainability-driven performance validation [
21] (see
Figure 3) show that clinically relevant visual indicators in the lungs were leveraged in the decision-making process. This validation is very important for auditing COVID-Net CXR-S in a transparent and responsible manner to ensure that not only is it leveraging the right indicators for driving the severity assessment process, but also that it is not primarily leveraging erroneous visual indicators (e.g., embedded markers, motion artifacts, imaging artifacts, etc.) to make ’right decisions for the wrong reasons’. Furthermore, this validation has the potential to help in the discovery of additional visual indicators to assist a clinician in their severity assessment as well as improve the trust that clinicians may have during operational use.
3.3. Radiologist Analysis
The expert radiologist findings and observations for select patient cases with respect to the identified critical factors during explainability-driven performance validation as shown in
Figure 3 are as follows. In all three cases, COVID-Net CXR-S correctly detected them to be patients with Level 2 airspace severity, which were clinically confirmed.
Case 1. According to radiologist findings, it was observed by both radiologists that there is patchy airspace opacity in the lower left lung lobe, which is consistent with one of the identified critical factors leveraged by COVID-Net CXR-S;
Case 2. According to radiologist findings, it was observed by both radiologists that there are patchy airspace opacities in the left and right midlung regions that coincide with the identified critical factors leveraged by COVID-Net CXR-S in that region. It was further observed by one of the radiologists that there are additional lower lobe opacities in both lungs;
Case 3. According to radiologist findings, it was observed by one of the radiologists that there are hilar opacities on the right lung that coincide with the identified critical factors leveraged by COVID-Net CXR-S. It was also observed that there are opacities in the left lower lobe, with the superior aspect of the opacities being leveraged by COVID-Net CXR-S. The second radiologist observed patchy airspace opacities in the right lung that overlap with the critical factors leveraged by COVID-Net CXR-S.
As such, based on the radiologist findings and observations on the three patient cases, it was shown that although some opacities were not identified by GSInquire as critical factors driving the decision-making behavior of COVID-Net CXR-2, several other abnormalities identified as critical factors were consistent with radiologist interpretations. Therefore, based on the critical factors identified by GSInquire as critical factors driving the decision-making behavior of COVID-Net CXR-S, the network was able to differentiate between the airspace severity levels but not necessarily leverage all regions of concern in making its severity assessment decisions.
4. Conclusions
In this study, we introduced COVID-Net CXR-S, a convolutional neural network for the prediction of airspace severity of a SARS-CoV-2 positive patient based on a CXR image of the patient’s chest. By leveraging transfer learning, we transferred representational knowledge gained from over 16,000 CXR images from a multinational cohort of over 15,000 SARS-CoV-2 positive and negative patient cases into a custom network architecture for severity assessment. The promising quantitative and qualitative results obtained from the conducted experiments demonstrate that the proposed COVID-Net CXR-S, while not a production-ready solution, can be a potentially become a powerful tool for aiding clinicians and front-line healthcare workers via computer-aided severity assessment of CXR images of COVID-19 positive patients. The ultimate goal for the open source release of COVID-Net CXR-S (
https://www.covid-net.ml) is to act as a catalyst for clinical scientists, machine learning researchers, as well as citizen scientists to develop innovative new clinical decision support solutions for helping clinicians around the world manage the continuing pandemic.
Author Contributions
H.A., M.P., M.J.S. and A.W. conceived the experiments, H.A. and M.P. conducted the experiments, all authors analysed the results, A.A. and A.S. reviewed and reported on select patient cases and corresponding explainability results illustrating model’s decision-making behavior, and all authors reviewed the manuscript. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
The study was conducted according to the guidelines of the University of Waterloo, and has received ethics clearance from the University of Waterloo (42235) as of 27 May 2020.
Data Availability Statement
All datasets, data preperation and model scripts are available publicly in an open source manner at
https://www.covid-net.ml.
Acknowledgments
We thank the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada Research Chairs program, the Canadian Institute for Advanced Research (CIFAR), DarwinAI Corp., and the organizations and initiatives from around the world collecting valuable COVID-19 data to advance science and knowledge.
Conflicts of Interest
M.J.S. and A.W. are affiliated with DarwinAI Corp.
References
- Pak, A.; Adegboye, O.A.; Adekunle, A.I.; Rahman, K.M.; McBryde, E.S.; Eisen, D.P. Economic consequences of the COVID-19 outbreak: The need for epidemic preparedness. Front. Public Health 2020, 8, 241. [Google Scholar] [CrossRef] [PubMed]
- Azoulay, E.; Beloucif, S.; Guidet, B.; Pateron, D.; Vivien, B.; Le Dorze, M. Admission decisions to intensive care units in the context of the major COVID-19 outbreak: Local guidance from the COVID-19 Paris-region area. Crit. Care 2020, 24, 293. [Google Scholar] [CrossRef] [PubMed]
- Tyrrell, C.S.; Mytton, O.T.; Gentry, S.V.; Thomas-Meyer, M.; Allen, J.L.Y.; Narula, A.A.; McGrath, B.; Lupton, M.; Broadbent, J.; Ahmed, A.; et al. Managing intensive care admissions when there are not enough beds during the COVID-19 pandemic: A systematic review. Thorax 2021, 76, 302–312. [Google Scholar] [CrossRef] [PubMed]
- Wang, L.; Lin, Z.Q.; Wong, A. COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-ray Images. Sci. Rep. 2020, 10, 19549. [Google Scholar] [CrossRef] [PubMed]
- Jain, R.; Gupta, M.; Taneja, S.; Hemanth, D.J. Deep learning based detection and analysis of COVID-19 on chest X-ray images. Appl. Intell. 2021, 51, 1690–1700. [Google Scholar] [CrossRef] [PubMed]
- Sahlol, A.T.; Yousri, D.; Ewees, A.A.; Al-Qaness, M.A.; Damasevicius, R.; Abd Elaziz, M. COVID-19 image classification using deep features and fractional-order marine predators algorithm. Sci. Rep. 2020, 10, 15364. [Google Scholar]
- Gunraj, H.; Sabri, A.; Koff, D.; Wong, A. COVID-Net CT-2: Enhanced Deep Neural Networks for Detection of COVID-19 from Chest CT Images Through Bigger, More Diverse Learning. arXiv 2021, arXiv:2101.07433. [Google Scholar]
- Gunraj, H.; Wang, L.; Wong, A. COVIDNet-CT: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest CT Images. Front. Med. 2020, 7, 608525. [Google Scholar] [CrossRef] [PubMed]
- He, X.; Yang, X.; Zhang, S.; Zhao, J.; Zhang, Y.; Xing, E.; Xie, P. Sample-efficient deep learning for covid-19 diagnosis based on ct scans. MedRxiv 2020. [Google Scholar] [CrossRef]
- Zhang, K.; Liu, X.; Shen, J.; Li, Z.; Sang, Y.; Wu, X.; Zha, Y.; Liang, W.; Wang, C.; Wang, K.; et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell 2020, 181, 1423–1433. [Google Scholar] [CrossRef]
- Wong, A.; Lin, Z.Q.; Wang, L.; Chung, A.G.; Shen, B.; Abbasi, A.; Hoshmand-Kochi, M.; Duong, T.Q. COVIDNet-S: Towards computer-aided severity assessment via training and validation of deep neural networks for geographic extent and opacity extent scoring of chest X-rays for SARS-CoV-2 lung disease severity. arXiv 2020, arXiv:2005.12855. [Google Scholar]
- Cohen, J.P.; Dao, L.; Roth, K.; Morrison, P.; Bengio, Y.; Abbasi, A.F.; Shen, B.; Mahsa, H.K.; Ghassemi, M.; Li, H.; et al. Predicting covid-19 pneumonia severity on chest x-ray with deep learning. Cureus 2020, 12, e9448. [Google Scholar] [CrossRef] [PubMed]
- Wong, A.; Shafiee, M.J.; Chwyl, B.; Li, F. FermiNets: Learning generative machines to generate efficient neural networks via generative synthesis. arXiv 2018, arXiv:1809.05989. [Google Scholar]
- Wong, A. NetScore: Towards Universal Metrics for Large-scale Performance Analysis of Deep Neural Networks for Practical On-Device Edge Usage. arXiv 2018, arXiv:1806.05512. [Google Scholar]
- Radiological Society of North America. RSNA Pneumonia Detection Challenge. Available online: https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/data (accessed on 1 January 2021).
- Radiological Society of North America. COVID-19 Radiography Database. Available online: https://www.kaggle.com/tawsifurrahman/covid19-radiography-database (accessed on 1 January 2021).
- Chung, A. Figure 1 COVID-19 Chest X-ray Data Initiative. Available online: https://github.com/agchung/Figure1-COVID-chestxray-dataset (accessed on 1 January 2021).
- Chung, A. ActualMed COVID-19 Chest X-ray Data Initiative. Available online: https://github.com/agchung/Actualmed-COVID-chestxray-dataset (accessed on 1 January 2021).
- Cohen, J.P.; Morrison, P.; Dao, L. COVID-19 image data collection. arXiv 2020, arXiv:2003.11597. [Google Scholar]
- Tsai, E.B.; Simpson, S.; Lungren, M.; Hershman, M.; Roshkovan, L.; Colak, E.; Erickson, B.J.; Shih, G.; Stein, A.; Kalpathy-Cramer, J.; et al. The RSNA International COVID-19 Open Annotated Radiology Database (RICORD). Radiology 2021, 299, 203957. [Google Scholar] [CrossRef] [PubMed]
- Lin, Z.Q.; Shafiee, M.J.; Bochkarev, S.; Jules, M.S.; Wang, X.Y.; Wong, A. Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms. arXiv 2019, arXiv:1910.07387. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 30 June 2016; pp. 770–778. [Google Scholar]
- Rajpurkar, P.; Irvin, J.; Zhu, K.; Yang, B.; Mehta, H.; Duan, T.; Ding, D.; Bagul, A.; Langlotz, C.; Shpanskaya, K.; et al. CheXNet: Radiologist-Level Pneumonia Detection on Chest X-rays with Deep Learning. arXiv 2017, arXiv:1711.05225. [Google Scholar]
| Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).