[Paper Review] Dual-stream Network for Visual Recognition

Mingyuan Mao, Renrui Zhang+7 more|arXiv (Cornell University)|May 31, 2021

Advanced Neural Network Applications55 references36 citations

TL;DR

DS-Net introduces dual-stream blocks that separately process high-resolution local details and low-resolution global patterns, with inter-scale alignment to fuse them, achieving strong results on ImageNet and MSCOCO.

ABSTRACT

Transformers with remarkable global representation capacities achieve competitive results for visual tasks, but fail to consider high-level local pattern information in input images. In this paper, we present a generic Dual-stream Network (DS-Net) to fully explore the representation capacity of local and global pattern features for image classification. Our DS-Net can simultaneously calculate fine-grained and integrated features and efficiently fuse them. Specifically, we propose an Intra-scale Propagation module to process two different resolutions in each block and an Inter-Scale Alignment module to perform information interaction across features at dual scales. Besides, we also design a Dual-stream FPN (DS-FPN) to further enhance contextual information for downstream dense predictions. Without bells and whistles, the proposed DS-Net outperforms DeiT-Small by 2.4% in terms of top-1 accuracy on ImageNet-1k and achieves state-of-the-art performance over other Vision Transformers and ResNets. For object detection and instance segmentation, DS-Net-Small respectively outperforms ResNet-50 by 6.4% and 5.5% in terms of mAP on MSCOCO 2017, and surpasses the previous state-of-the-art scheme, which significantly demonstrates its potential to be a general backbone in vision tasks. The code will be released soon.

Motivation & Objective

• Motivate the need for jointly modeling local and global patterns in vision models.
• Propose a Dual-stream Network (DS-Net) that preserves dual-scale representations for improved recognition.
• Design Intra-scale Propagation to process dual resolutions and Inter-scale Alignment to fuse them.
• Extend the dual-stream design to Feature Pyramid Networks (DS-FPN) for dense prediction tasks.

Proposed method

• Introduce Dual-stream Blocks (DS-Blocks) that split features into local (high-res) and global (low-res) paths.
• Process local features with depth-wise 3x3 convolutions to capture fine-grained details.
• Process global features with self-attention on a downsampled token sequence to capture object-level relations.
• Apply Inter-scale Alignment using co-attention to bidirectionally fuse local and global representations.
• Attach DS-Blocks to FPN to form DS-FPN for enhanced multi-scale context.

Experimental results

Research questions

• RQ1 Can a dual-stream architecture effectively disentangle local and global visual information for classification and dense prediction tasks?
• RQ2 Do intra-scale propagation and inter-scale alignment improve cross-scale feature fusion over naive fusion methods?
• RQ3 Does DS-FPN provide measurable gains for object detection and instance segmentation compared to standard FPN?
• RQ4 What is the impact of the local/global feature ratio on performance?

Key findings

• DS-Net outperforms DeiT-Small by 2.4% top-1 accuracy on ImageNet-1k.
• DS-Net variants achieve competitive or state-of-the-art results among Vision Transformers and CNNs on ImageNet classification.
• On MSCOCO 2017, DS-Net-S* improves RetinaNet with ResNet-50 by 6.4% APbbox and Mask R-CNN by 6.1% APbbox.
• DS-Net-S* achieves 40.2% AP for instance segmentation, surpassing ResNet-50 and Swin-T by 5.5% and 0.4% respectively.
• DS-Net-T* and DS-Net-S* show additional gains over their non-aligned counterparts, demonstrating the benefit of Inter-scale Alignment.
• DS-FPN with DS-Blocks provides mAP improvements over standard FPN for both object detection and instance segmentation.