[Paper Review] ESFPNet: efficient deep learning architecture for real-time lesion segmentation in autofluorescence bronchoscopic video

Chang Qi, Danish Ahmad+3 more|arXiv (Cornell University)|Jul 15, 2022

Lung Cancer Diagnosis and Treatment4 citations

TL;DR

ESFPNet is a real-time deep learning architecture for automatic bronchial lesion segmentation in autofluorescence bronchoscopy (AFB) video, leveraging pretrained Mix Transformer (MiT) encoders and an efficient stage-wise feature pyramid (ESFP) decoder. It achieves a mean Dice score of 0.756 and processes video at 27 frames per second, enabling practical real-time clinical use.

ABSTRACT

Lung cancer tends to be detected at an advanced stage, resulting in a high patient mortality rate. Thus, much recent research has focused on early disease detection Bronchoscopy is the procedure of choice for an effective noninvasive way of detecting early manifestations (bronchial lesions) of lung cancer. In particular, autofluorescence bronchoscopy (AFB) discriminates the autofluorescence properties of normal (green) and diseased tissue (reddish brown) with different colors. Because recent studies show AFB's high sensitivity in searching lesions, it has become a potentially pivotal method in bronchoscopic airway exams. Unfortunately, manual inspection of AFB video is extremely tedious and error prone, while limited effort has been expended toward potentially more robust automatic AFB lesion analysis. We propose a real-time (processing throughput of 27 frames/sec) deep-learning architecture dubbed ESFPNet for accurate segmentation and robust detection of bronchial lesions in AFB video streams. The architecture features an encoder structure that exploits pretrained Mix Transformer (MiT) encoders and an efficient stage-wise feature pyramid (ESFP) decoder structure. Segmentation results from the AFB airway-exam videos of 20 lung cancer patients indicate that our approach gives a mean Dice index = 0.756 and an average Intersection of Union = 0.624, results that are superior to those generated by other recent architectures. Thus, ESFPNet gives the physician a potential tool for confident real-time lesion segmentation and detection during a live bronchoscopic airway exam. Moreover, our model shows promising potential applicability to other domains, as evidenced by its state-of-the-art (SOTA) performance on the CVC-ClinicDB, ETIS-LaribPolypDB datasets, and superior performance on the Kvasir, CVC-ColonDB datasets.

Motivation & Objective

• Address the critical need for early lung cancer detection by automating lesion identification in autofluorescence bronchoscopy (AFB) video.
• Overcome limitations of manual AFB video review, which is tedious, error-prone, and time-consuming.
• Develop a real-time, accurate, and robust deep learning model for segmenting bronchial lesions in AFB video streams.
• Ensure the model generalizes well across diverse medical imaging domains beyond bronchoscopy.
• Achieve high inference speed without sacrificing segmentation accuracy, enabling use during live bronchoscopy.

Proposed method

• Adopt a U-Net-inspired encoder-decoder architecture with a pretrained Mix Transformer (MiT) encoder for hierarchical feature extraction.
• Design an efficient stage-wise feature pyramid (ESFP) decoder to enhance feature aggregation across multi-scale contexts.
• Integrate deep supervision to improve feature learning and segmentation accuracy at multiple decoder stages.
• Utilize lightweight, parameter-efficient components to maintain high inference speed (27 FPS) while preserving performance.
• Train the model end-to-end on a limited AFB dataset of 208 lesion frames from 20 lung cancer patients.
• Apply transfer learning and domain adaptation techniques to improve generalization on external polyp segmentation datasets.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve real-time, accurate segmentation of bronchial lesions in autofluorescence bronchoscopy video?
• RQ2 How does the proposed ESFPNet architecture compare to state-of-the-art models in terms of segmentation accuracy and inference speed?
• RQ3 To what extent can the ESFPNet model generalize to other medical image segmentation tasks, such as colonic polyp detection?
• RQ4 Does the stage-wise feature pyramid decoder effectively enhance feature representation compared to conventional pyramid structures?
• RQ5 Can the model maintain high performance despite limited training data from clinical AFB video?

Key findings

• ESFPNet achieved a mean Dice score of 0.756 and mean Intersection over Union (mIoU) of 0.624 on the AFB lung cancer patient dataset, outperforming other recent architectures.
• The model processed video at 27 frames per second, enabling real-time application during live bronchoscopic procedures.
• On the CVC-ClinicDB dataset, ESFPNet-L achieved a mDice of 0.949 and mIoU of 0.907, ranking first among all compared models.
• On the Kvasir-SEG dataset, ESFPNet-L achieved a mDice of 0.931 and mIoU of 0.887, ranking second-best among all models tested.
• In generalizability tests, ESFPNet-L achieved mDice of 0.827 and mIoU of 0.752 on ETIS-LaribPolypDB, and mDice of 0.823 and mIoU of 0.741 on CVC-ColonDB.
• The model demonstrated strong performance across diverse datasets, indicating high generalization and learning ability in medical image segmentation tasks.