[Paper Review] Deep High-Resolution Representation Learning for Visual Recognition

Jingdong Wang, Ke Sun+10 more|arXiv (Cornell University)|Aug 20, 2019

Advanced Neural Network Applications186 references351 citations

TL;DR

High-Resolution Network (HRNet) maintains high-resolution representations throughout all stages by connecting multi-resolution streams in parallel and repeatedly exchanging information across resolutions, achieving strong results across pose estimation, semantic segmentation, and object detection.

ABSTRACT

High-resolution representations are essential for position-sensitive vision problems, such as human pose estimation, semantic segmentation, and object detection. Existing state-of-the-art frameworks first encode the input image as a low-resolution representation through a subnetwork that is formed by connecting high-to-low resolution convolutions \emph{in series} (e.g., ResNet, VGGNet), and then recover the high-resolution representation from the encoded low-resolution representation. Instead, our proposed network, named as High-Resolution Network (HRNet), maintains high-resolution representations through the whole process. There are two key characteristics: (i) Connect the high-to-low resolution convolution streams \emph{in parallel}; (ii) Repeatedly exchange the information across resolutions. The benefit is that the resulting representation is semantically richer and spatially more precise. We show the superiority of the proposed HRNet in a wide range of applications, including human pose estimation, semantic segmentation, and object detection, suggesting that the HRNet is a stronger backbone for computer vision problems. All the codes are available at~{\url{https://github.com/HRNet}}.

Motivation & Objective

• Motivate the need for position-sensitive vision representations where high spatial detail is crucial.
• Propose a network architecture that preserves high-resolution representations throughout processing rather than recovering from low-resolution features.
• Show that parallel multi-resolution streams with repeated cross-resolution fusion yield semantically rich and spatially precise representations.
• Demonstrate HRNet's superiority across multiple vision tasks such as pose estimation, semantic segmentation, and object detection.

Proposed method

• Initialize with a high-resolution stem and progressively add high-to-low resolution streams in parallel.
• Maintain four parallel resolution streams and perform repeated multi-resolution fusion to exchange information across resolutions.
• Define three representation heads (HRNetV1, HRNetV2, HRNetV2+p) to output high-resolution or multi-resolution fused representations.
• Instantiate HRNet with four stages containing multi-resolution branches of increasing depth and channel widths, with residual units and batch normalization.
• Establish fusion transforms that downsample or upsample between resolutions to produce consistent multi-resolution representations.

Experimental results

Research questions

• RQ1 Can preserving high-resolution representations throughout the network improve performance on position-sensitive vision tasks?
• RQ2 How do parallel multi-resolution streams and repeated cross-resolution fusion compare to traditional downsample-then-restore backbones for vision tasks?
• RQ3 What are the effects of different HRNet head designs (V1, V2, V2+p) on performance across pose estimation, segmentation, and detection?
• RQ4 Do HRNet-based backbones provide state-of-the-art results across semantic segmentation and object detection benchmarks with comparable model size and compute?

Key findings

• HRNet achieves strong pose estimation performance on COCO with a high-resolution backbone outperforming prior methods while maintaining lower or comparable computational cost.
• HRNetV2 and HRNetV2+p variants yield state-of-the-art results on semantic segmentation benchmarks such as Cityscapes, PASCAL-Context, and LIP with similar model sizes and lower computation.
• The four-resolution parallel design with repeated fusion strengthens high- and mid-level representations, leading to improved localization and segmentation quality across tasks.
• HRNet-based detectors (HRNetV2p) integrated into standard detection frameworks (e.g., Faster R-CNN, Cascade R-CNN, FCOS, CenterNet) improve detection performance, especially for small objects.
• The paper provides comprehensive comparisons showing HRNet surpasses prior architectures that rely on recovering high-resolution representations from low-resolution backbones across multiple datasets and tasks.