[Paper Review] TESL-Net: A Transformer-Enhanced CNN for Accurate Skin Lesion Segmentation

Shahzaib Iqbal, Muhammad Zeeshan+3 more|arXiv (Cornell University)|Aug 19, 2024

Cutaneous Melanoma Detection and ManagementMedicine3 citations

TL;DR

TESL-Net proposes a hybrid deep learning architecture combining a CNN encoder-decoder with Swin Transformer blocks and Bi-ConvLSTM modules to improve skin lesion segmentation. By integrating global context via transformers and sequential modeling via Bi-ConvLSTM in skip connections, it achieves state-of-the-art performance, outperforming SOTA methods by up to 11.22% in Jaccard index across ISIC 2017 and ISIC 2018 datasets.

ABSTRACT

Early detection of skin cancer relies on precise segmentation of dermoscopic images of skin lesions. However, this task is challenging due to the irregular shape of the lesion, the lack of sharp borders, and the presence of artefacts such as marker colours and hair follicles. Recent methods for melanoma segmentation are U-Nets and fully connected networks (FCNs). As the depth of these neural network models increases, they can face issues like the vanishing gradient problem and parameter redundancy, potentially leading to a decrease in the Jaccard index of the segmentation model. In this study, we introduced a novel network named TESL-Net for the segmentation of skin lesions. The proposed TESL-Net involves a hybrid network that combines the local features of a CNN encoder-decoder architecture with long-range and temporal dependencies using bi-convolutional long-short-term memory (Bi-ConvLSTM) networks and a Swin transformer. This enables the model to account for the uncertainty of segmentation over time and capture contextual channel relationships in the data. We evaluated the efficacy of TESL-Net in three commonly used datasets (ISIC 2016, ISIC 2017, and ISIC 2018) for the segmentation of skin lesions. The proposed TESL-Net achieves state-of-the-art performance, as evidenced by a significantly elevated Jaccard index demonstrated by empirical results.

Motivation & Objective

• Address the challenge of precise skin lesion segmentation in dermoscopic images due to irregular shapes, blurry borders, and artifacts like hair and marker colors.
• Overcome limitations of deep CNNs, such as vanishing gradients and parameter redundancy, which degrade segmentation performance.
• Enhance feature representation by integrating long-range dependencies and temporal modeling into a U-Net-like architecture.
• Improve segmentation accuracy on benchmark datasets (ISIC 2016, 2017, 2018) through a novel fusion of CNN, Swin Transformer, and Bi-ConvLSTM components.
• Achieve state-of-the-art performance in Jaccard index, Dice score, and other metrics across multiple public skin lesion segmentation benchmarks.

Proposed method

• Employ a U-Net-style encoder-decoder architecture as the backbone for feature learning and segmentation.
• Integrate Swin Transformer blocks in the encoder to capture long-range contextual relationships and global image dependencies.
• Insert Bi-ConvLSTM modules into skip connections to model sequential and spatial-temporal dependencies across feature maps.
• Use multi-scale feature fusion and skip connections to preserve spatial details while enhancing semantic representation.
• Leverage multi-head self-attention mechanisms in the Swin Transformer to dynamically weigh feature importance across local and global receptive fields.
• Train the end-to-end model using binary cross-entropy and Dice loss to optimize segmentation performance on skin lesion boundaries.

Figure 1: Challenges in skin lesion segmentation.

Experimental results

Research questions

• RQ1 Can a hybrid architecture combining CNNs, Transformers, and Bi-ConvLSTM improve skin lesion segmentation accuracy compared to standard U-Net and SOTA models?
• RQ2 How effective is the integration of Swin Transformer blocks in capturing long-range contextual features in dermoscopic images with complex textures and artifacts?
• RQ3 To what extent do Bi-ConvLSTM modules in skip connections enhance feature representation by modeling temporal and spatial dependencies across feature maps?
• RQ4 Does the proposed TESL-Net architecture generalize across diverse skin lesion datasets with varying lesion morphology, contrast, and artifact presence?
• RQ5 Can the fusion of local (CNN), global (Transformer), and sequential (Bi-ConvLSTM) modeling lead to superior performance on standard evaluation metrics like Jaccard index and Dice score?

Key findings

• TESL-Net achieved a Jaccard index (IoU) of 86.91% on the ISIC 2017 dataset, outperforming the previous SOTA method (RA-Net) by 2.02%.
• On the ISIC 2018 dataset, TESL-Net achieved a Jaccard index of 90.56%, surpassing the next-best method (RA-Net) by 2.22%.
• The model achieved a Dice score of 94.22% on ISIC 2018, demonstrating high overlap between predicted and ground-truth segmentation masks.
• In the ISIC 2016 benchmark, TESL-Net achieved a Jaccard index 8.13% higher than the SOTA method, confirming consistent superiority across datasets.
• Visual comparisons showed that TESL-Net effectively handles challenging cases such as irregular lesion shapes, low contrast, hair, and ink artifacts.
• The model achieved a specificity of 97.29% on ISIC 2017 and 97.21% on ISIC 2018, indicating strong performance in correctly identifying healthy skin regions.

Figure 2: Schematic of the proposed method. (a) Block diagram of the proposed TESL-Net, (b) Swin Transformer Block.