[Paper Review] Compact Generalized Non-local Network

Kaiyu Yue, Ming Sun+4 more|arXiv (Cornell University)|Oct 31, 2018

Neural Networks and ApplicationsComputer Science26 references91 citations

TL;DR

CGNL extends non-local networks to model cross-channel position interactions with a compact Taylor-based approximation, yielding improved performance on fine-grained classification and action recognition across multiple datasets.

ABSTRACT

The non-local module is designed for capturing long-range spatio-temporal dependencies in images and videos. Although having shown excellent performance, it lacks the mechanism to model the interactions between positions across channels, which are of vital importance in recognizing fine-grained objects and actions. To address this limitation, we generalize the non-local module and take the correlations between the positions of any two channels into account. This extension utilizes the compact representation for multiple kernel functions with Taylor expansion that makes the generalized non-local module in a fast and low-complexity computation flow. Moreover, we implement our generalized non-local method within channel groups to ease the optimization. Experimental results illustrate the clear-cut improvements and practical applicability of the generalized non-local module on both fine-grained object recognition and video classification. Code is available at: https://github.com/KaiyuYue/cgnl-network.pytorch.

Motivation & Objective

• Motivate modeling interactions between positions across channels to improve fine-grained recognition.
• Propose a compact generalized non-local (CGNL) module with a Taylor-series based kernel approximation for efficiency.
• Incorporate channel grouping to ease optimization and scalability.
• Demonstrate CGNL effectiveness on fine-grained classification and action recognition tasks.
• Provide insights and ablations comparing CGNL to original non-local blocks and baselines.

Proposed method

• Generalize non-local operation to model correlations between any positions across channels (Eq. 8).
• Rewrite the non-local computation as vec(Y)=f(vec(XW_theta),vec(XW_phi)) vec(XW_g) with a more expressive f.
• Approximate pairwise kernel f via Taylor expansion to obtain a compact representation (Eq. 12).
• Implement grouped CGNL blocks to manage capacity and optimization by splitting channels into G groups.
• Use a residual-like block structure with CGNL inside standard CNN backbones (Eq. 13).
• Compare kernel choices (dot product, embedded Gaussian, Gaussian RBF) and show dot product is often best in practice.

Experimental results

Research questions

• RQ1 Can modeling long-range dependencies across channels (not just positions) improve recognition of fine-grained objects and actions?
• RQ2 Does a compact, Taylor-expanded CGNL provide similar accuracy gains as full non-local while reducing computational cost?
• RQ3 How do channel groupings affect optimization and performance of the CGNL block?
• RQ4 Which kernel function best approximates the generalized non-local interaction in practice?
• RQ5 Do CGNL blocks improve performance across image and video tasks (CUB, Mini-Kinetics, UCF101, COCO) compared to standard NL blocks?

Key findings

• CGNL consistently improves over the original non-local block across tasks (e.g., fine-grained classification and action recognition).
• Dot-product kernel often yields the best performance among evaluated kernels for CGNL.
• Channel grouping can boost performance up to an optimal group count (e.g., 8 groups on Mini-Kinetics) but too many groups can hinder cross-channel correlations.
• CGNL achieves larger gains than NL blocks when added with comparable numbers of blocks (e.g., 1 or 5 blocks).
• On ImageNet, adding 1 CGNL block improves top-1 accuracy over baselines (e.g., R-50 from 76.15 to 77.69).
• CGNL blocks improve Mask R-CNN results on COCO over the NL blocks when added to the backbone.