[Paper Review] Learning Human-Object Interactions by Graph Parsing Neural Networks

Siyuan Qi, Wenguan Wang+3 more|arXiv (Cornell University)|Aug 23, 2018

Multimodal Machine Learning ApplicationsComputer Science39 references59 citations

TL;DR

GPNN learns parse graphs and performs end-to-end message passing to detect and recognize human-object interactions in images and videos, outperforming state-of-the-art on HOI benchmarks. It jointly infers graph structure and propagates messages for HOI tasks in both static and temporal settings.

ABSTRACT

This paper addresses the task of detecting and recognizing human-object interactions (HOI) in images and videos. We introduce the Graph Parsing Neural Network (GPNN), a framework that incorporates structural knowledge while being differentiable end-to-end. For a given scene, GPNN infers a parse graph that includes i) the HOI graph structure represented by an adjacency matrix, and ii) the node labels. Within a message passing inference framework, GPNN iteratively computes the adjacency matrices and node labels. We extensively evaluate our model on three HOI detection benchmarks on images and videos: HICO-DET, V-COCO, and CAD-120 datasets. Our approach significantly outperforms state-of-art methods, verifying that GPNN is scalable to large datasets and applies to spatial-temporal settings. The code is available at https://github.com/SiyuanQi/gpnn.

Motivation & Objective

• Motivate a unified, end-to-end framework that explicitly represents HOI structures as graphs.
• Develop a differentiable graph parsing neural network to infer parse graphs and perform HOI labeling.
• Demonstrate scalability of GPNN to large HOI datasets and applicability to spatial-temporal HOI tasks.

Proposed method

• Represent HOI as a complete graph with human and object nodes and learnable adjacency for parse graph derivation.
• Define four modular functions—Link, Message, Update, Readout—for end-to-end differentiable graph parsing and belief-propagation-like inference.
• Iteratively update adjacency and node states across S steps to jointly learn graph structure and HOI labels.
• Implement Link with a neural network producing A via 1x1 convolutions; use convLSTM for temporal graphs.
• Use GRU-based updates and a readout network to produce HOI action/object labels.
• Train with losses on adjacency structure (L1) and node outputs (multi-label hinge loss).

Experimental results

Research questions

• RQ1 Can GPNN jointly learn the parse graph structure and perform HOI labeling in a differentiable, end-to-end manner?
• RQ2 Does learning an adaptive graph structure improve HOI detection and recognition over fixed-graph or purely neural baselines?
• RQ3 Is the framework scalable to large HOI datasets and applicable to spatial-temporal HOI tasks?
• RQ4 How does iterative graph learning (multiple message-passing steps) affect performance?
• RQ5 What is the impact of supervision on the learned graph (adjacency) versus fixed or unregularized graphs?

Key findings

• GPNN achieves substantial improvements over baselines on HOI detection in images (HICO-DET and V-COCO) and HOI recognition/anticipation in videos (CAD-120).
• On HICO-DET, GPNN reaches notable gains over previous methods across Full, Rare, and Non-Rare categories.
• On V-COCO, GPNN outperforms baselines across Set 1, Set 2, and Average mAP.
• In CAD-120, GPNN achieves higher sub-activity and object affordance detection F1-scores and anticipates sub-activities better than ATCRF and S-RNN.
• Ablation studies show that learning graph structure, joint parsing with message passing, and iterative refinement all contribute to performance gains.