[Paper Review] HRFormer: High-Resolution Transformer for Dense Prediction

Yuhui Yuan, Rao Fu+5 more|arXiv (Cornell University)|Oct 18, 2021

Human Pose and Action RecognitionComputer Science58 references126 citations

TL;DR

HRFormer combines high-resolution multi-scale representations with local-window self-attention and FFN depth-wise convolution to enable efficient dense prediction, achieving strong pose estimation and semantic segmentation results while using fewer parameters and FLOPs than comparable HRNet and transformer baselines.

ABSTRACT

We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer by $1.3$ AP on COCO pose estimation with $50\%$ fewer parameters and $30\%$ fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer.

Motivation & Objective

• Motivate dense prediction with high-resolution representations beyond traditional ViT-style low-resolution outputs.
• Design a transformer block that preserves high-resolution streams while enabling cross-scale information exchange.
• Reduce memory and computation using local-window self-attention and a convolution-augmented FFN.
• Integrate HRNet-style multi-resolution fusion to maintain parallel high- and low-resolution streams.
• Demonstrate effectiveness on pose estimation, semantic segmentation, and ImageNet classification.

Proposed method

• Adopts a multi-resolution parallel architecture inspired by HRNet to maintain high-resolution representations across stages.
• Implements local-window self-attention within non-overlapping KxK image windows to reduce memory and compute from quadratic to linear in spatial size.
• Introduces a 3x3 depth-wise convolution in the FFN to enable information exchange across non-overlapping windows and to expand receptive field.
• Uses relative position embeddings in local-window self-attention to incorporate spatial structure.
• Utilizes convolutional multi-scale fusion modules for cross-resolution information exchange, following HRNet’s fusion design.
• Arrays transformer blocks across four resolution streams, with four-stage architecture and window sizes fixed at (7,7,7,7) by default.

Experimental results

Research questions

• RQ1 Can high-resolution, multi-scale transformer representations improve dense prediction tasks such as pose estimation and semantic segmentation?
• RQ2 Does combining local-window self-attention with a depth-wise convolution in the FFN provide competitive accuracy with lower memory and compute vs. full global attention?
• RQ3 How does a HRNet-inspired multi-resolution fusion strategy interact with transformer blocks to boost performance across tasks?
• RQ4 What are the impacts of design choices (window size, FFN structure) on accuracy and efficiency across Vision tasks?

Key findings

• HRFormer-B achieves 0.9% AP improvement over HRNet-W48 with 32% fewer parameters and 19% fewer FLOPs on COCO val pose estimation.
• HRFormer-B + OCR improves HRNet-W48 + OCR by 1.2% mIoU on PASCAL-Context and 2.0% mIoU on COCO-Stuff with 25% fewer parameters and similar FLOPs.
• On ImageNet-1K, HRFormer-B surpasses DeiT-B by 1.0% top-1 accuracy while using ~40% fewer parameters and ~20% fewer FLOPs.
• HRFormer-T/S/B outperform HRNet counterparts and competitive transformer baselines across pose estimation and segmentation tasks, with notably lower parameter and compute costs.
• Ablation shows 3x3 depth-wise convolution in FFN significantly improves ImageNet, PASCAL-Context, and COCO metrics compared to FFN without DW conv.
• Compared to ViT/DeiT/Swin baselines on pose estimation, HRFormer-B achieves better results with fewer parameters and FLOPs.