Evaluating Generalizability of Deep Learning Models Using Indian-COVID-19 CT Dataset
Computer tomography (CT) have been routinely used for the diagnosis of lung
diseases and recently, during the pandemic, for detecting the infectivity and
severity of COVID-19 disease. One of the major concerns in using ma-chine
learning (ML) approaches for automatic processing of CT scan images in clinical
setting is that these methods are trained on limited and biased sub-sets of
publicly available COVID-19 data. This has raised concerns regarding the
generalizability of these models on external datasets, not seen by the model
during training. To address some of these issues, in this work CT scan images
from confirmed COVID-19 data obtained from one of the largest public
repositories, COVIDx CT 2A were used for training and internal vali-dation of
machine learning models. For the external validation we generated
Indian-COVID-19 CT dataset, an open-source repository containing 3D CT volumes
and 12096 chest CT images from 288 COVID-19 patients from In-dia. Comparative
performance evaluation of four state-of-the-art machine learning models, viz.,
a lightweight convolutional neural network (CNN), and three other CNN based
deep learning (DL) models such as VGG-16, ResNet-50 and Inception-v3 in
classifying CT images into three classes, viz., normal, non-covid pneumonia,
and COVID-19 is carried out on these two datasets. Our analysis showed that the
performance of all the models is comparable on the hold-out COVIDx CT 2A test
set with 90% - 99% accuracies (96% for CNN), while on the external
Indian-COVID-19 CT dataset a drop in the performance is observed for all the
models (8% - 19%). The traditional ma-chine learning model, CNN performed the
best on the external dataset (accu-racy 88%) in comparison to the deep learning
models, indicating that a light-weight CNN is better generalizable on unseen
data. The data and code are made available at https://github.com/aleesuss/c19.