[Paper Review] Learning Human-Object Interaction Detection using Interaction Points

Tiancai Wang, Tong Yang+4 more|arXiv (Cornell University)|Mar 31, 2020

Multimodal Machine Learning ApplicationsComputer Science53 references35 citations

TL;DR

The paper presents a fully convolutional, anchor-free HOI detector that frames interaction modeling as keypoint detection of interaction points plus grouping with human/object detections to predict HOI triplets.

ABSTRACT

Understanding interactions between humans and objects is one of the fundamental problems in visual classification and an essential step towards detailed scene understanding. Human-object interaction (HOI) detection strives to localize both the human and an object as well as the identification of complex interactions between them. Most existing HOI detection approaches are instance-centric where interactions between all possible human-object pairs are predicted based on appearance features and coarse spatial information. We argue that appearance features alone are insufficient to capture complex human-object interactions. In this paper, we therefore propose a novel fully-convolutional approach that directly detects the interactions between human-object pairs. Our network predicts interaction points, which directly localize and classify the inter-action. Paired with the densely predicted interaction vectors, the interactions are associated with human and object detections to obtain final predictions. To the best of our knowledge, we are the first to propose an approach where HOI detection is posed as a keypoint detection and grouping problem. Experiments are performed on two popular benchmarks: V-COCO and HICO-DET. Our approach sets a new state-of-the-art on both datasets. Code is available at https://github.com/vaesl/IP-Net.

Motivation & Objective

• Motivate the need to move beyond appearance-based, instance-centric HOI detection architectures that scale poorly with many human-object pairs.
• Introduce a new HOI representation based on interaction points and interaction vectors to directly localize and classify interactions.
• Develop a fully-convolutional network that detects interaction points and vectors and groups them with detected humans/objects to form HOI triplets.
• Demonstrate state-of-the-art performance on two benchmarks (V-COCO and HICO-DET) with ablations validating each component.

Proposed method

• Pose HOI detection as a keypoint detection and grouping problem inspired by anchor-free object detection.
• Use an Hourglass backbone to extract features and produce two parallel branches: interaction point heatmaps and unsigned interaction vector maps.
• Train interaction point heatmaps with Gaussian supervision and use focal-like loss to balance positives/negatives.
• Train interaction vectors to predict the absolute horizontal/vertical lengths toward human/object centers (unsigned vector).
• At inference, extract top-k interaction points, recover interaction vectors, and form interaction boxes.
• Group interaction points with detected human/object boxes via a soft-constraint scheme that checks IoU with human/object boxes and corner distances to a reference box.

Experimental results

Research questions

• RQ1 Can HOI detection be effectively formulated as a keypoint detection and grouping problem rather than a multimodal, instance-centric pipeline?
• RQ2 Do interaction points and vectors improve HOI localization and classification compared to traditional multi-stream approaches?
• RQ3 What is the impact of the proposed interaction grouping and auxiliary components (angle-filter, dist-ratio-filter, center-pool) on HOI detection performance?
• RQ4 Are the proposed method and components scalable and effective on standard HOI benchmarks (V-COCO, HICO-DET)?

Key findings

• The proposed IP-Net achieves state-of-the-art mAP_role on V-COCO (51.0 without pre-training on HICO-DET, 52.3 with HICO-DET pretraining) and on HICO-DET under both Default and Known Object settings.
• Ablation studies show the interaction grouping scheme with the interaction box and corner-distance constraints substantially improves performance (e.g., 46.2 to 50.5 to 51.0 mAP_role on V-COCO).
• Center-pool and the two-branch interaction generation (point heatmaps and unsigned vectors) provide measurable gains over baselines, with a total absolute improvement of 11.4 percentage points over the baseline.
• The method maintains efficient grouping with near-linear complexity in practice (<5 ms) despite quadratic-looking formulation in theory, due to filtering via heatmaps and soft constraints.
• Dynamic thresholding for interaction scores improves rare vs. non-rare class performance on HICO-DET compared to fixed thresholds.