[Paper Review] COVID-19 Pneumonia Severity Prediction using Hybrid Convolution-Attention Neural Architectures

Nam V. Nguyen, Chang Jm|arXiv (Cornell University)|Jul 6, 2021

COVID-19 diagnosis using AI16 references4 citations

TL;DR

This paper proposes a hybrid convolution-attention neural network with a data-centric pre-training framework to improve COVID-19 pneumonia severity prediction from limited medical imaging data. By leveraging self-attention from Transformers and Dense Associative Memory (Hopfield networks), the model achieves R² = 0.85 ± 0.05 and ρ = 0.92 ± 0.02 in geographic extent prediction, significantly outperforming transfer learning baselines.

ABSTRACT

This study proposed a novel framework for COVID-19 severity prediction, which is a combination of data-centric and model-centric approaches. First, we propose a data-centric pre-training for extremely scare data scenarios of the investigating dataset. Second, we propose two hybrid convolution-attention neural architectures that leverage the self-attention from the Transformer and the Dense Associative Memory (Modern Hopfield networks). Our proposed approach achieves significant improvement from the conventional baseline approach. The best model from our proposed approach achieves $R^2 = 0.85 \pm 0.05$ and Pearson correlation coefficient $ρ= 0.92 \pm 0.02$ in geographic extend and $R^2 = 0.72 \pm 0.09, ρ= 0.85\pm 0.06$ in opacity prediction.

Motivation & Objective

• Address the challenge of low-performance deep learning models in COVID-19 severity prediction due to extremely small and scarce datasets.
• Develop a data-centric pre-training framework to enhance representation learning for limited medical imaging data.
• Design hybrid convolution-attention architectures that integrate self-attention mechanisms to capture global contextual information in medical images.
• Improve predictive performance for both geographic extent and opacity severity metrics in COVID-19 pneumonia using novel neural architectures.
• Demonstrate that data-centric pre-training and hybrid attention mechanisms outperform conventional transfer learning from ImageNet.

Proposed method

• Propose a data-centric pre-training framework that generates synthetic or augmented data to improve feature learning before downstream severity prediction.
• Integrate self-attention modules from the Transformer architecture and Dense Associative Memory (modern Hopfield network) into convolutional neural network backbones.
• Use hybrid architectures (HCT and HCH) that combine convolutional feature extraction with global context modeling via attention mechanisms.
• Train the models in two stages: first, pre-train on the pre-processing data using the data-centric framework; second, fine-tune on the severity prediction task.
• Employ cross-validation with loss computed via Equation 6 to evaluate model generalization and training stability.
• Utilize backbone models including DenseNet121, ResNet50, and EfficientNet-B4, with weights initialized from the data-centric pre-training phase.

Experimental results

Research questions

• RQ1 Can a data-centric pre-training framework significantly improve the performance of deep learning models in low-data regimes for COVID-19 severity prediction?
• RQ2 How do hybrid convolution-attention architectures incorporating self-attention and Hopfield networks compare to standard CNNs and transfer learning baselines in predicting pneumonia severity?
• RQ3 To what extent does the integration of global context modeling via attention mechanisms enhance predictive accuracy for geographic extent and opacity in COVID-19 imaging?
• RQ4 Does the proposed framework yield consistent improvements across different backbone architectures, especially in small-scale datasets?
• RQ5 What is the impact of data-centric pre-training on model convergence and generalization, as measured by cross-validation loss and correlation metrics?

Key findings

• The proposed data-centric pre-training framework significantly improves model performance over ImageNet transfer learning, with a notable increase in R² and Pearson correlation across all backbone models.
• The best-performing model, HCT-DenseNet121, achieved R² = 0.85 ± 0.05 and ρ = 0.92 ± 0.02 in geographic extent prediction, demonstrating high predictive accuracy.
• Opacity prediction also improved substantially, with R² = 0.72 ± 0.09 and ρ = 0.85 ± 0.06, showing strong alignment between actual and predicted values.
• Hybrid architectures reduced mean squared error (MSE) by approximately 0.2 points and mean absolute error (MAE) by 1.5 points on average compared to standalone CNNs.
• The cross-validation loss of hybrid models was consistently lower than standalone CNNs, indicating better generalization and training stability.
• The performance gain was most pronounced in DenseNet121, with R² increasing from 0.74 to 0.82 in geographic extent prediction after hybrid attention integration.