[Paper Review] Selective Feature Connection Mechanism: Concatenating Multi-layer CNN Features with a Feature Selector

Chen Du, Chunheng Wang+3 more|arXiv (Cornell University)|Nov 15, 2018

Advanced Neural Network Applications55 references4 citations

TL;DR

This paper proposes the Selective Feature Connection Mechanism (SFCM), a lightweight, end-to-end trainable module that selectively fuses low-level and high-level CNN features using a feature selector generated from high-level features. By mimicking human visual attention, SFCM improves performance across image classification, scene text detection, and image-to-image translation with minimal computational overhead and consistent gains over baselines.

ABSTRACT

Different layers of deep convolutional neural networks(CNNs) can encode different-level information. High-layer features always contain more semantic information, and low-layer features contain more detail information. However, low-layer features suffer from the background clutter and semantic ambiguity. During visual recognition, the feature combination of the low-layer and high-level features plays an important role in context modulation. If directly combining the high-layer and low-layer features, the background clutter and semantic ambiguity may be caused due to the introduction of detailed information. In this paper, we propose a general network architecture to concatenate CNN features of different layers in a simple and effective way, called Selective Feature Connection Mechanism (SFCM). Low-level features are selectively linked to high-level features with a feature selector which is generated by high-level features. The proposed connection mechanism can effectively overcome the above-mentioned drawbacks. We demonstrate the effectiveness, superiority, and universal applicability of this method on multiple challenging computer vision tasks, including image classification, scene text detection, and image-to-image translation.

Motivation & Objective

• To address the challenge of combining low-level detail features with high-level semantic features in deep CNNs, which often introduces noise and ambiguity due to background clutter.
• To develop a general-purpose, lightweight module that enables selective fusion of multi-layer CNN features without degrading semantic representation capacity.
• To improve performance in diverse computer vision tasks—such as image classification, scene text detection, and image-to-image translation—by effectively modulating feature flow with attention.
• To design a mechanism that aligns with biological visual attention, where high-level semantics guide the selection of relevant low-level features.

Proposed method

• SFCM introduces a feature selector generated from high-level features to dynamically control which low-level features are concatenated, enabling attention-based fusion.
• The method supports two connection modes: direct connection and residual connection, both of which are differentiable and trainable via standard backpropagation.
• The feature selector is implemented as a learnable attention map that attends to spatial locations in low-level feature maps based on high-level context.
• The mechanism is plug-and-play, easily integrated into existing CNN architectures such as U-Net, EAST, and pix2pix without architectural overhaul.
• The feature selector is trained end-to-end with the main network, using standard optimization (e.g., Adam), with minimal parameter and computation overhead.
• The approach leverages the complementary strengths of low-level features (detail) and high-level features (semantics), reducing noise from irrelevant low-level signals.

Experimental results

Research questions

• RQ1 Can a learnable, high-level-driven feature selector improve the fusion of multi-level CNN features in visual recognition tasks?
• RQ2 Does selective feature connection outperform direct concatenation of multi-scale features in terms of accuracy and robustness?
• RQ3 Can SFCM be universally applied across diverse computer vision tasks, including detection and image translation?
• RQ4 How does SFCM affect the model's ability to detect small or multi-oriented text in cluttered scenes?
• RQ5 To what extent does SFCM enhance the quality of generated images in image-to-image translation, particularly in preserving fine details?

Key findings

• On the ICDAR 2015 scene text detection benchmark, EAST with SFCM achieved an F-score of 0.8254, significantly improving both precision and recall over the baseline.
• On the COCO-Text dataset, the SFCM-enhanced EAST model achieved an F-score of 0.4554, demonstrating improved detection of multi-oriented and small text instances.
• In image-to-image translation on the Cityscapes dataset, the cGAN with SFCM achieved a per-pixel accuracy of 0.71, up from 0.66 in the baseline, with better detail preservation in generated images.
• Qualitative results show that SFCM-generated images have sharper edges and more accurate structural details, especially in complex scenes like urban landscapes.
• The SFCM consistently improved performance across all evaluated tasks, confirming its generalizability and effectiveness in diverse deep learning pipelines.
• The method adds minimal parameters and computation, making it suitable for deployment in resource-constrained environments.