[Paper Review] Non-local Neural Networks

Xiaolong Wang, Ross Girshick+2 more|arXiv (Cornell University)|Nov 21, 2017

Human Pose and Action RecognitionComputer Science55 references147 citations

TL;DR

Introduces non-local operations as a generic building block to capture long-range dependencies in vision models, showing strong improvements in video classification and COCO tasks, with competitive results to state-of-the-art without heavy bells and whistles.

ABSTRACT

Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, even without any bells and whistles, our non-local models can compete or outperform current competition winners on both Kinetics and Charades datasets. In static image recognition, our non-local models improve object detection/segmentation and pose estimation on the COCO suite of tasks. Code is available at https://github.com/facebookresearch/video-nonlocal-net .

Motivation & Objective

• Motivate the need for modeling long-range dependencies beyond local convolutions and recurrent connections in vision tasks.
• Propose a generic non-local operation that computes responses as a weighted sum over all positions to capture global context.
• Demonstrate that non-local blocks can improve video classification, object detection/segmentation, and pose estimation with modest computational overhead.

Proposed method

• Define a generic non-local operation y_i = (1/C(x)) sum_j f(x_i, x_j) g(x_j) where f measures pairwise relation and g embeds the input at position j.
• Instantiate non-local blocks with variants of f (Gaussian, embedded Gaussian, dot-product, concatenation) and g (linear embedding via 1x1 or 1x1x1 convolutions).
• Wrap the operation in a residual non-local block z_i = W_z y_i + x_i to enable easy integration into pre-trained networks.
• Use bottleneck design (reducing channels in W_g, W_theta, W_phi) and optional subsampling to reduce computation.
• Incorporate non-local blocks into 2D and inflated 3D CNNs (C2D and I3D) for video models, and test on COCO for detection/segmentation and pose estimation.

Experimental results

Research questions

• RQ1 Can non-local operators directly model long-range dependencies in space, time, or spacetime, and how do they compare to traditional local convolutions or recurrent approaches?
• RQ2 Are non-local blocks efficient and beneficial even when added in small numbers or at different stages of a backbone?
• RQ3 Do non-local blocks complement 3D convolutions in video models and extend to static image tasks like COCO detection/segmentation and keypoint estimation?

Key findings

• A single non-local block can yield about 1% improvement on Kinetics when added to a 2D baseline.
• Embedded Gaussian, dot-product, and concatenation variants perform similarly, with embedded Gaussian used by default for interpretability.
• Adding multiple non-local blocks yields further gains; e.g., 5-block and 10-block configurations improve top-1 accuracy on Kinetics, with spacetime Non-local nets outperforming some deeper baselines.
• Non-local blocks applied in spacetime (space and time jointly) provide larger gains than space-only or time-only variants.
• Non-local nets outperform the RGB I3D baseline on Kinetics/Charades and can be more FLOP-efficient than 3D convolutions in some settings.
• On COCO, a single non-local block improves AP^box and AP^mask across backbones, with small computational overhead, and additional blocks yield diminishing returns.