[Paper Review] Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey

Guoping Xu, Xiaxia Wang+3 more|arXiv (Cornell University)|May 2, 2024

Brain Tumor Detection and Classification10 citations

TL;DR

This survey chronicles the origin, development, and impact of skip connections and residual learning in deep neural networks for computer vision and medical imaging, outlining backbone architectures, techniques, and future directions.

ABSTRACT

Deep learning has made significant progress in computer vision, specifically in image classification, object detection, and semantic segmentation. The skip connection has played an essential role in the architecture of deep neural networks,enabling easier optimization through residual learning during the training stage and improving accuracy during testing. Many neural networks have inherited the idea of residual learning with skip connections for various tasks, and it has been the standard choice for designing neural networks. This survey provides a comprehensive summary and outlook on the development of skip connections in deep neural networks. The short history of skip connections is outlined, and the development of residual learning in deep neural networks is surveyed. The effectiveness of skip connections in the training and testing stages is summarized, and future directions for using skip connections in residual learning are discussed. Finally, we summarize seminal papers, source code, models, and datasets that utilize skip connections in computer vision, including image classification, object detection, semantic segmentation, and image reconstruction. We hope this survey could inspire peer researchers in the community to develop further skip connections in various forms and tasks and the theory of residual learning in deep neural networks. The project page can be found at https://github.com/apple1986/Residual_Learning_For_Images

Motivation & Objective

• Trace the short history and origin of skip connections and residual learning.
• Summarize how skip connections influence training and testing performance in deep nets.
• Review important backbone architectures that inherit residual learning (e.g., ResNet, DenseNet) and their applications in CV/medical imaging.
• Discuss theoretical insights and future directions for residual learning and skip connections across tasks and architectures.

Proposed method

• Survey the historical development from negative feedback to residual learning and skip connections.
• Categorize residual learning advances by skip-length, block widening, discriminative feature learning, efficiency, and self-attention integration.
• Summarize key architectures and modules that employ skip connections (e.g., ResNet, DenseNet, Inception variants, ResNeXt, Res2Net, WRN, DenseNet).
• Discuss techniques to strengthen residual learning (activation order, attention blocks like SENet/SKNet/ResNeSt, bottleneck redesigns, dilated and deformable convolutions).
• Examine efficiency-focused modifications (1x1, group, depthwise convolutions, stochastic depth, pruning, distillation) and self-attention frameworks (Transformers, ViT, Swin Transformer) that use skip connections.

Experimental results

Research questions

• RQ1 What are the origins and evolution of skip connections in deep neural networks?
• RQ2 How do skip connections affect optimization and generalization in training deep CV/medical imaging models?
• RQ3 What architectural variants (long vs short skip, widened blocks, multi-branch designs) maximize performance and efficiency?
• RQ4 What roles do attention mechanisms and self-attention-based models play in residual learning?
• RQ5 What resources (datasets, code, models) demonstrate skip-connection-based approaches across CV and medical imaging?

Key findings

• Skip connections/residual learning significantly ease optimization and improve final accuracy in deep networks.
• Long skip connections (e.g., FCN, U-Net) effectively fuse detailed spatial information with high-level semantics for segmentation and related tasks.
• Widening residual blocks and multi-branch designs (DenseNet, ResNeXt, Res2Net, WRN) can boost performance and efficiency beyond depth-only increases.
• Attention modules (SENet, SKNet, ResNeSt, CBAM) integrated with residual blocks enhance discriminative feature learning.
• Advanced convolutional techniques (dilated, deformable, depthwise, group, 1x1) and stochastic depth/pruning strategies improve efficiency without sacrificing accuracy.
• Transformers and self-attention (ViT, Swin Transformer) leverage skip connections and remain central to modern CV architectures and pretraining paradigms.