[Paper Review] A hybrid deep learning framework for Covid-19 detection via 3D Chest CT Images

Shuang Liang|arXiv (Cornell University)|Jul 8, 2021

COVID-19 diagnosis using AI16 references4 citations

TL;DR

This paper proposes CTNet, a hybrid deep learning framework combining 3D CNN with self-attention mechanisms via transformers to detect COVID-19 from chest CT scans. By introducing a data resampling strategy that preserves lesion continuity and reduces input size, CTNet achieves a macro F1 score of 88.23% on the COV19-CT-DB validation set—18.23 percentage points above the baseline—while maintaining fast inference at 380ms per case.

ABSTRACT

In this paper, we present a hybrid deep learning framework named CTNet which combines convolutional neural network and transformer together for the detection of COVID-19 via 3D chest CT images. It consists of a CNN feature extractor module with SE attention to extract sufficient features from CT scans, together with a transformer model to model the discriminative features of the 3D CT scans. Compared to previous works, CTNet provides an effective and efficient method to perform COVID-19 diagnosis via 3D CT scans with data resampling strategy. Advanced results on a large and public benchmarks, COV19-CT-DB database was achieved by the proposed CTNet, over the state-of-the-art baseline approachproposed together with the dataset.

Motivation & Objective

• To address the challenge of high computational cost and unstable training in 3D CT-based COVID-19 detection due to large input volumes.
• To reduce the size of 3D CT scans without losing discriminative lesion features by proposing a novel data resampling strategy.
• To improve diagnostic performance by combining CNNs for local feature extraction with transformers for long-range dependency modeling.
• To develop an efficient, end-to-end trainable framework suitable for clinical deployment with limited computational resources.
• To outperform existing state-of-the-art methods on the public COV19-CT-DB dataset in terms of both accuracy and inference speed.

Proposed method

• A data resampling strategy is proposed to uniformly sample a fixed number of CT slices (set to 32) from each 3D scan, reducing input size while preserving lesion continuity.
• The input module resamples CT scans using a hybrid resampling-oversampling approach based on total scan count and desired resampled number (renumCT).
• A 3D CNN feature extractor with squeeze-and-excitation (SE) attention modules enhances feature discrimination by recalibrating channel-wise feature responses.
• The extracted features are flattened and fed into two parallel branches: a transformer-based information aggregation module and a fully connected (FC) branch.
• Feature fusion is performed via element-wise addition (EWA) between the transformer and FC branches before final classification using a softmax layer.
• The model is trained end-to-end using stochastic gradient descent with a step-decay learning rate schedule on two 1080Ti GPUs.

Experimental results

Research questions

• RQ1 Can a data resampling strategy effectively reduce the computational burden of 3D CT-based COVID-19 detection while preserving clinically relevant features?
• RQ2 Does combining CNN-based local feature extraction with transformer-based global context modeling improve diagnostic performance on 3D CT scans?
• RQ3 How does the proposed hybrid CTNet framework compare to existing state-of-the-art methods in terms of macro F1 score and inference speed?
• RQ4 To what extent does the proposed framework reduce the need for full-volume CT processing, enabling faster and more efficient clinical deployment?
• RQ5 Is the model robust to variable input sizes and capable of generalizing across diverse 3D CT scans from multiple hospitals?

Key findings

• The proposed data resampling strategy successfully reduces input size from up to 700 slices to 32 while maintaining diagnostic relevance.
• CTNet achieved a macro F1 score of 88.23% on the COV19-CT-DB validation set, representing a 18.23 percentage point improvement over the baseline method.
• The FC-only branch of CTNet achieved a macro F1 score of 87.96%, demonstrating strong performance even without transformer fusion.
• The full CTNet (fused) model outperformed both the baseline and the FC-only variant, confirming the effectiveness of transformer-based feature aggregation.
• Inference time for each case was only 380ms, indicating strong practical feasibility for real-time clinical use.
• Ablation studies confirmed that the combination of resampling and hybrid architecture leads to superior performance with reduced computational demand.