[Paper Review] Interact as You Intend: Intention-Driven Human-Object Interaction Detection

Bingjie Xu, Junnan Li+3 more|arXiv (Cornell University)|Aug 29, 2018

Multimodal Machine Learning Applications46 references16 citations

TL;DR

This paper proposes iHOI, a novel intention-driven framework for human-object interaction (HOI) detection that leverages human gaze and body pose to improve detection accuracy. By modeling attention-guided contextual regions and relative joint distances to objects, and using a hard negative triplet mining strategy, iHOI achieves state-of-the-art performance on V-COCO and HICO-DET, with +4.28 mAP gain on V-COCO when jointly modeling gaze and pose.

ABSTRACT

The recent advances in instance-level detection tasks lay strong foundation for genuine comprehension of the visual scenes. However, the ability to fully comprehend a social scene is still in its preliminary stage. In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications. To undertake social tasks interacting with objects, humans direct their attention and move their body based on their intention. Based on this observation, we provide a unique computational perspective to explore human intention in HOI detection. Specifically, the proposed human intention-driven HOI detection (iHOI) framework models human pose with the relative distances from body joints to the object instances. It also utilizes human gaze to guide the attended contextual regions in a weakly-supervised setting. In addition, we propose a hard negative sampling strategy to address the problem of mis-grouping. We perform extensive experiments on two benchmark datasets, namely V-COCO and HICO-DET. The efficacy of each proposed component has also been validated.

Motivation & Objective

• To improve human-object interaction detection by modeling human intention through observable visual cues such as gaze and body posture.
• To address the challenge of mis-grouping in HOI detection by introducing a hard negative triplet mining strategy.
• To enhance detection robustness and accuracy by leveraging contextually relevant regions guided by human gaze.
• To provide a computational framework that integrates intention-aware visual reasoning into instance-level HOI detection.
• To validate the effectiveness of intention modeling in improving both correct prediction confidence and false positive suppression.

Proposed method

• The framework models human intention using two key components: gaze-guided attention to select informative contextual regions and relative joint-to-object distances to encode pose-based intention.
• It employs a weakly-supervised setting to utilize gaze predictions, dynamically selecting top-5 most relevant scene regions based on gaze heatmaps.
• Body pose is encoded as relative distances from human joints to object instances, capturing spatial relationships indicative of intention.
• A novel hard negative triplet mining strategy targets mis-grouped HOI candidates by focusing on ambiguous or hard-to-classify triplets.
• The model is trained end-to-end on V-COCO and HICO-DET, with joint optimization of HOI triplet classification and intention modeling.
• The framework uses a two-stream architecture to process human and object features, with cross-attention mechanisms to align intention cues with interaction predictions.

Figure 1 : An example where the actor’s intention is informative of the HOI $\langle\mathsf{person},\mathsf{hold},\mathsf{cup}\rangle$ . The intention is represented using the attended regions and body pose. Specifically, he is fixating at the regions around the cup that he is interacting with, and

Experimental results

Research questions

• RQ1 Can human gaze and body posture serve as reliable, computationally exploitable cues for intention modeling in HOI detection?
• RQ2 How does integrating intention modeling improve the accuracy and robustness of HOI detection in complex social scenes?
• RQ3 To what extent can gaze-guided contextual region selection enhance detection performance compared to global scene features?
• RQ4 Does hard negative triplet mining based on mis-grouping risk improve generalization and reduce hallucination in HOI prediction?
• RQ5 Can intention modeling suppress false positives caused by ambiguous actions or incorrect object localization?

Key findings

• Joint modeling of gaze and pose yields a +4.28 mAP improvement on V-COCO and +1.42 mAP on HICO-DET over the baseline VTransE-HO model.
• Using top-5 gaze-guided contextual regions achieves optimal performance, balancing robustness to gaze errors and noise reduction.
• Incorporating relative joint-to-object distances improves mAP by +0.80 on V-COCO and +0.53 on HICO-DET compared to coordinate-based pose encoding.
• The proposed hard negative triplet mining strategy reduces mAP drop by 3.42 points on V-COCO compared to no mining, significantly suppressing interaction hallucinations.
• Qualitative results show that intention modeling effectively suppresses false positives—e.g., rejecting 'kicking' when pose and gaze do not align with the target object.
• The model successfully rejects incorrect triplets like 'eating another person’s cake' when intention cues (gaze and pose) contradict the action.

Figure 2 : The proposed iHOI takes an image as input for feature extraction and human intention-driven interaction prediction, and outputs the detected triplets. Human intention is modelled as follows: 1) the pose information is incorporated with the distances from body joints to the instance center