[Paper Review] LatentGNN: Learning Efficient Non-local Relations for Visual Recognition

Songyang Zhang, Shipeng Yan+1 more|arXiv (Cornell University)|May 28, 2019

Advanced Neural Network Applications47 references46 citations

TL;DR

LatentGNN introduces a latent-space graph neural network to model non-local feature relations with a low-rank adjacency representation, enabling scalable, context-aware feature augmentation for visual recognition tasks. It yields large performance gains with reduced computation compared to prior non-local methods.

ABSTRACT

Capturing long-range dependencies in feature representations is crucial for many visual recognition tasks. Despite recent successes of deep convolutional networks, it remains challenging to model non-local context relations between visual features. A promising strategy is to model the feature context by a fully-connected graph neural network (GNN), which augments traditional convolutional features with an estimated non-local context representation. However, most GNN-based approaches require computing a dense graph affinity matrix and hence have difficulty in scaling up to tackle complex real-world visual problems. In this work, we propose an efficient and yet flexible non-local relation representation based on a novel class of graph neural networks. Our key idea is to introduce a latent space to reduce the complexity of graph, which allows us to use a low-rank representation for the graph affinity matrix and to achieve a linear complexity in computation. Extensive experimental evaluations on three major visual recognition tasks show that our method outperforms the prior works with a large margin while maintaining a low computation cost.

Motivation & Objective

• Capture long-range dependencies in visual feature representations to enhance recognition tasks.
• Provide a scalable, efficient non-local context modeling mechanism compatible with standard ConvNets.
• Learn task-driven non-local relations through a low-rank, mixture-of-kernel graph representation.
• Demonstrate effectiveness across multiple vision tasks (object detection, segmentation, and point cloud segmentation).

Proposed method

• Augment convolutional features with a latent graph network that introduces latent nodes connected to feature nodes and to each other.
• Perform three-step message passing: visible-to-latent, latent-to-latent, and latent-to-visible to estimate non-local context.
• Represent the full graph affinity as a low-rank mixture of kernels via latent variables, enabling linear-time computation.
• Provide a matrix-form interpretation showing the latent GNN corresponds to a low-rank approximation of the dense affinity matrix.
• Allow stacking multiple latent GNN layers in deep ConvNets and fusing with original features through a residual-like augmentation.
• Extend to both grid-like feature graphs for 2D vision tasks and irregular graphs for 3D point clouds; validate with end-to-end training on task-specific losses.

Experimental results

Research questions

• RQ1 Can a latent-variable GNN achieve efficient and flexible modeling of non-local relations in visual features?
• RQ2 Does a low-rank, mixture-of-kernel representation improve scalability and performance over fully-connected GNNs in vision tasks?
• RQ3 Can LatentGNN be integrated into standard detectors/segmenters and point cloud networks to boost accuracy with modest computational overhead?
• RQ4 How does LatentGNN perform across both 2D image-based and 3D point cloud segmentation tasks?
• RQ5 What is the impact of adding multiple latent layers or multiple low-rank kernels on performance and cost?

Key findings

• LatentGNN consistently improves object detection and instance segmentation performance over baselines while keeping computation cost low.
• A latent space with a small number of latent nodes enables linear-time message passing, yielding substantial speedups over dense affinity-based GNNs.
• A mixture of multiple low-rank kernels further enhances expressive power and performance compared to a single kernel.
• Incorporating LatentGNN at various backbone stages yields notable gains, with multi-stage deployment providing larger improvements.
• LatentGNN also benefits irregular graph data, achieving improvements in 3D point cloud semantic segmentation tasks.
• The framework is modular and end-to-end trainable, and supports stacking multiple LatentGNN layers within standard CNN backbones.