[Paper Review] Detecting and Recognizing Human-Object Interactions

Georgia Gkioxari, Ross Girshick+2 more|arXiv (Cornell University)|Apr 24, 2017

Multimodal Machine Learning Applications28 references60 citations

TL;DR

The paper introduces InteractNet, a Faster R-CNN based model with a human-centric branch that predicts action-specific target-object locations to detect and recognize <human, verb, object> triplets in images. It achieves state-of-the-art role AP on V-COCO and strong results on HICO-DET with efficient, end-to-end training.

ABSTRACT

To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with. To exploit this cue, our model learns to predict an action-specific density over target object locations based on the appearance of a detected person. Our model also jointly learns to detect people and objects, and by fusing these predictions it efficiently infers interaction triplets in a clean, jointly trained end-to-end system we call InteractNet. We validate our approach on the recently introduced Verbs in COCO (V-COCO) and HICO-DET datasets, where we show quantitatively compelling results.

Motivation & Objective

• Motivate and tackle human-object interaction recognition as <human, verb, object> triplets in real-world images.
• Leverage human appearance (pose, action) to predict likely target object locations and reduce search space.
• Jointly train an end-to-end system that fuses human-centric action cues with standard object detection and pairwise interaction reasoning.
• Demonstrate effectiveness on V-COCO and HICO-DET datasets and achieve practical inference speed.

Proposed method

• Extend Faster R-CNN with a human-centric branch that classifies actions and predicts a target object location density per action.
• Model the target object location as a 4-d Gaussian mean μ_h^a conditioned on the human appearance and action, with g_h,o^a as the likelihood term combining b_o with μ_h^a.
• Compute triplet score S_h,o^a = s_h · s_o · s_h^a · g_h,o^a and perform cascaded inference to keep O(n) runtime complexity.
• Optionally replace s_h^a with an interaction branch s_{h,o}^a that combines human and object appearance for action scoring.
• Train all branches jointly as a multi-task objective including object detection, action classification, and target localization losses.
• Inference identifies, for each detected human and action, the object that maximizes s_o · s_{h,o}^a · g_h,o^a and forms the triplet <human, verb, object>.

Experimental results

Research questions

• RQ1 Can a human-centric cue improve localization of the target object involved in a human-action, thereby improving triplet detection accuracy?
• RQ2 Does jointly training object detection, action classification, and target localization in a single end-to-end framework improve interaction recognition performance?
• RQ3 How does the proposed target localization density (uni-modal vs. multi-modal) affect detection accuracy for various actions?
• RQ4 What is the effect of an optional interaction branch that conditions action scores on both human and object appearances?

Key findings

• InterActNet achieves AP_role of 40.0 on all actions on V-COCO test, an absolute improvement of 8.2 points over the strong baseline (31.8).
• InterActNet improves AP_role by 26% relatively over the baseline on V-COCO (from 31.8 to 40.0).
• On the HICO-DET dataset, InteractNet outperforms the prior approach by approximately 27% relative improvement.
• The ablations show target localization is the key contributor to performance; removing it reduces AP_role to 31.9 (from 37.5).
• The method runs at ~135 ms per image on a single Nvidia M40 GPU, indicating practical efficiency.
• Using FPN backbones significantly boosts performance over vanilla ResNet-50, especially for small objects.