[Paper Review] CARAFE: Content-Aware ReAssembly of FEatures

Jiaqi Wang, Kai Chen+4 more|arXiv (Cornell University)|May 6, 2019

Advanced Neural Network Applications47 references41 citations

TL;DR

CARAFE is a universal, lightweight upsampling operator that uses content-aware, region-based kernel reassembly to improve dense prediction tasks with large receptive fields.

ABSTRACT

Feature upsampling is a key operation in a number of modern convolutional network architectures, e.g. feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose Content-Aware ReAssembly of FEatures (CARAFE), a universal, lightweight and highly effective operator to fulfill this goal. CARAFE has several appealing properties: (1) Large field of view. Unlike previous works (e.g. bilinear interpolation) that only exploit sub-pixel neighborhood, CARAFE can aggregate contextual information within a large receptive field. (2) Content-aware handling. Instead of using a fixed kernel for all samples (e.g. deconvolution), CARAFE enables instance-specific content-aware handling, which generates adaptive kernels on-the-fly. (3) Lightweight and fast to compute. CARAFE introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation and inpainting. CARAFE shows consistent and substantial gains across all the tasks (1.2%, 1.3%, 1.8%, 1.1db respectively) with negligible computational overhead. It has great potential to serve as a strong building block for future research. It has great potential to serve as a strong building block for future research. Code and models are available at https://github.com/open-mmlab/mmdetection.

Motivation & Objective

• Motivate and design a feature upsampling operator that can aggregate large contextual information for dense prediction tasks.
• Enable instance-specific (content-aware) handling during upsampling instead of fixed, global kernels.
• Maintain efficiency and lightweight computation while providing large receptive fields.
• Demonstrate universal effectiveness across object detection, instance/semantic segmentation, and image inpainting.

Proposed method

• Predict location-specific reassembly kernels from local content using a lightweight kernel-prediction module.
• Reassemble features within a predefined region via a weighted sum using the predicted kernels.
• Normalize kernels with softmax so the reassembly is a local, sum-to-1 weighting scheme.
• Integrate CARAFE into existing architectures by replacing standard upsampling operators (nearest/bilinear) or deconvolution.
• Analyze hyper-parameters such as encoder/kernel sizes and compressed channels to balance performance and efficiency.

Experimental results

Research questions

• RQ1 Can content-aware, region-based upsampling improve performance on dense prediction tasks beyond fixed upsampling methods?
• RQ2 What are the effects of kernel size, encoder size, and channel compression on CARAFE’s accuracy and efficiency?
• RQ3 Does CARAFE provide consistent gains across object detection, instance segmentation, semantic segmentation, and image inpainting?

Key findings

• CARAFE yields consistent improvements across tasks: +1.2% AP for Faster R-CNN, +1.3% AP for Mask R-CNN, +1.8% mIoU for ADE20k, and +1.1 dB PSNR for Places inpainting.
• It offers large receptive-field aggregation with content-aware, instance-specific kernels.
• The computational overhead is small compared to deconvolution, with CARAFE’s FLOPs substantially lower for upsampling a HxW feature map (2x) with 256 channels.
• Replacing standard upsampling in FPN and mask heads with CARAFE achieves the best AP in multiple ablations.
• Ablations show that using a compressed channel (C_m=64) maintains performance with efficiency gains; k_encoder≈k_up−2 is a good default; softmax normalization is effective.