[Paper Review] High-Resolution Representations for Labeling Pixels and Regions

Ke Sun, Yang Zhao+8 more|arXiv (Cornell University)|Apr 9, 2019

Digital Image Processing TechniquesComputer Science129 references661 citations

TL;DR

The paper extends HRNet by aggregating representations from all parallel high-to-low resolutions (HRNetV2), delivering stronger high-resolution features that achieve state-of-the-art results in semantic segmentation and facial landmark detection, and strong multi-level representations for object detection.

ABSTRACT

High-resolution representation learning plays an essential role in many vision problems, e.g., pose estimation and semantic segmentation. The high-resolution network (HRNet)~\cite{SunXLW19}, recently developed for human pose estimation, maintains high-resolution representations through the whole process by connecting high-to-low resolution convolutions in \emph{parallel} and produces strong high-resolution representations by repeatedly conducting fusions across parallel convolutions. In this paper, we conduct a further study on high-resolution representations by introducing a simple yet effective modification and apply it to a wide range of vision tasks. We augment the high-resolution representation by aggregating the (upsampled) representations from all the parallel convolutions rather than only the representation from the high-resolution convolution as done in~\cite{SunXLW19}. This simple modification leads to stronger representations, evidenced by superior results. We show top results in semantic segmentation on Cityscapes, LIP, and PASCAL Context, and facial landmark detection on AFLW, COFW, $300$W, and WFLW. In addition, we build a multi-level representation from the high-resolution representation and apply it to the Faster R-CNN object detection framework and the extended frameworks. The proposed approach achieves superior results to existing single-model networks on COCO object detection. The code and models have been publicly available at \url{https://github.com/HRNet}.

Motivation & Objective

• Motivate and improve high-resolution representations for pixel/region labeling tasks beyond pose estimation.
• Investigate a simple modification to HRNet to utilize representations from all parallel resolutions.
• Demonstrate the method across semantic segmentation, facial landmark detection, and object detection tasks.
• Show that multi-level, high-resolution features enhance detection of small objects and overall performance.

Proposed method

• Maintain high-resolution representations via parallel multi-resolution convolutions with repeated multi-scale fusion.
• Introduce HRNetV2 by aggregating upsampled representations from all parallel resolutions instead of only the high-resolution stream.
• Upsample and concatenate low-resolution branch features to form a richer high-resolution representation.
• For detection, downsample the high-resolution representation to create multi-level features for a feature pyramid (HRNetV2 p).
• Instantiate with a four-stage backbone and multi-resolution blocks; mix features from all resolutions before task-specific heads.
• Apply segmentation heads to high-resolution outputs for semantic segmentation and facial landmark heatmaps, and construct multi-level features for Faster R-CNN/Mask R-CNN/Cascade R-CNN.

Experimental results

Research questions

• RQ1 Can aggregating representations from all high-to-low resolution branches improve the quality of high-resolution features?
• RQ2 Do HRNetV2 representations yield better semantic segmentation and facial landmark detection than the original HRNet?
• RQ3 Can multi-level HRNet representations enhance object detection frameworks like Faster R-CNN and its extended variants?

Key findings

• HRNetV2 substantially strengthens high-resolution representations by leveraging all parallel resolutions.
• Achieves state-of-the-art results on Cityscapes, PASCAL Context, and LIP for semantic segmentation with efficient model size and computation.
• Attains best results on AFLW, COFW, 300W, and WFLW for facial landmark detection.
• Multi-level HRNet representations (HRNetV2 p) improve COCO object detection when integrated into Faster R-CNN/Mask R-CNN/Cascade R-CNN.
• Outperforms comparable single-model detectors on COCO test-dev without multi-scale training/testing in Faster R-CNN and Cascade R-CNN settings.