[Paper Review] iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection

Chen Gao, Yuliang Zou+1 more|arXiv (Cornell University)|Aug 30, 2018

Multimodal Machine Learning ApplicationsComputer Science180 citations

TL;DR

The paper proposes iCAN, an instance-centric attention module that learns to attend to informative regions conditioned on each detected instance, improving HOI detection on V-COCO and HICO-DET with state-of-the-art results.

ABSTRACT

Recent years have witnessed rapid progress in detecting and recognizing individual object instances. To understand the situation in a scene, however, computers need to recognize how humans interact with surrounding objects. In this paper, we tackle the challenging task of detecting human-object interactions (HOI). Our core idea is that the appearance of a person or an object instance contains informative cues on which relevant parts of an image to attend to for facilitating interaction prediction. To exploit these cues, we propose an instance-centric attention module that learns to dynamically highlight regions in an image conditioned on the appearance of each instance. Such an attention-based network allows us to selectively aggregate features relevant for recognizing HOIs. We validate the efficacy of the proposed network on the Verb in COCO and HICO-DET datasets and show that our approach compares favorably with the state-of-the-arts.

Motivation & Objective

• Motivate HOI detection as recognizing interactions between humans and surrounding objects for scene understanding.
• Leverage instance-specific appearance to guide spatially-aware contextual attention.
• Integrate a three-stream network (human, object, pairwise) with an instance-centric attention module to improve HOI prediction.
• Demonstrate state-of-the-art performance on V-COCO and HICO-DET and provide ablations and error analyses.

Proposed method

• Introduce an instance-centric attention module that creates a context feature by attending to image regions conditioned on an instance’s appearance.
• Embed instance appearance and convolutional features into a 512-channel space to compute a similarity-based attentional map.
• Extract contextual features as a weighted average of convolutional features using the attentional map and concatenate with instance appearance.
• Use three streams (human, object, pairwise) to produce action scores s_h^a, s_o^a, and s_sp^a, then fuse via multiplication/addition as S_h,o^a = s_h · s_o · (s_h^a + s_o^a) · s_sp^a.
• Adopt a late fusion variant (sums of stream scores) and an early fusion variant (concatenated features fed through FC layers) for HOI score prediction.
• Train as a multi-label classification problem with sigmoid outputs for each action.

Experimental results

Research questions

• RQ1 Does instance-centric attention conditioned on person/object appearance improve HOI prediction beyond appearance/spatial cues?
• RQ2 How do different fusion strategies (late vs early) affect HOI detection performance and efficiency?
• RQ3 What is the impact of context features (instance-centric attention) versus other contextual baselines on HOI accuracy?
• RQ4 Can the proposed iCAN framework achieve state-of-the-art results on V-COCO and HICO-DET datasets?

Key findings

• iCAN with late fusion achieves 44.7 AP_role on V-COCO, outperforming InteractNet (40.0 AP_role).
• iCAN with early fusion achieves 45.3 AP_role on V-COCO, improving over late fusion.
• On HICO-DET (default setting), iCAN achieves 14.84 Full, 10.45 Rare, and 16.15 Non Rare; Known Object updates are 16.26 Full, 11.33 Rare, 17.73 Non Rare.
• Compared to prior best methods, iCAN shows a 49.3% relative improvement on HICO-DET Default Full results over InteractNet.
• Ablation studies show instance-centric attention provides larger gains than other contextual features (e.g., bottom-up attention, full-image features).
• Qualitative results illustrate diverse HOI detections and interpretable attention maps highlighting relevant regions and body parts.