[Paper Review] Relational Context Learning for Human-Object Interaction Detection

Sanghyun Kim, Deunsol Jung+1 more|arXiv (Cornell University)|Apr 11, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes MUREN, a three-branch transformer-based method for human-object interaction (HOI) detection that enhances relational reasoning through multiplex relation context learning. By modeling unary, pairwise, and ternary relations among human, object, and interaction tokens, and using an attentive fusion module to selectively propagate context, MUREN achieves state-of-the-art performance on HICO-DET and V-COCO benchmarks.

ABSTRACT

Recent state-of-the-art methods for HOI detection typically build on transformer architectures with two decoder branches, one for human-object pair detection and the other for interaction classification. Such disentangled transformers, however, may suffer from insufficient context exchange between the branches and lead to a lack of context information for relational reasoning, which is critical in discovering HOI instances. In this work, we propose the multiplex relation network (MUREN) that performs rich context exchange between three decoder branches using unary, pairwise, and ternary relations of human, object, and interaction tokens. The proposed method learns comprehensive relational contexts for discovering HOI instances, achieving state-of-the-art performance on two standard benchmarks for HOI detection, HICO-DET and V-COCO.

Motivation & Objective

• Address the limited context exchange between decoder branches in existing two-branch transformer-based HOI detection methods.
• Overcome the lack of relational reasoning capacity due to insufficient interaction-aware context in disentangled architectures.
• Improve HOI detection by modeling rich, multi-level relational context (unary, pairwise, ternary) between human, object, and interaction tokens.
• Design a three-branch architecture to learn more discriminative features for human detection, object detection, and interaction classification.
• Enable effective and task-specific context exchange between branches through an attentive fusion mechanism.

Proposed method

• Propose a multiplex relation embedding module (MURE) that computes unary, pairwise, and ternary relation contexts from human, object, and interaction tokens.
• Model unary context as self-attention within each token type (e.g., interaction-only context), pairwise context as cross-attention between two types (e.g., human-interaction), and ternary context as joint representation of all three.
• Introduce an attentive fusion module that selects relevant context information for each sub-task using task-specific token conditioning and channel-wise attention.
• Use element-wise addition to fuse task-specific tokens with selected relation context for each branch, enabling targeted context exchange.
• Adopt a three-branch transformer decoder architecture: one for human detection, one for object detection, and one for interaction classification.
• Train end-to-end with a standard HOI detection loss, optimizing for detection and classification heads via multi-task learning.

Figure 1 : The illustration of relation context information in an HOI instance. We define three types of relation context information in an HOI instance: unary, pairwise, and ternary relation contexts. Each relation context provides useful information for detecting an HOI instance. For example, in o

Experimental results

Research questions

• RQ1 Can modeling multiple levels of relational context (unary, pairwise, ternary) improve HOI detection performance?
• RQ2 Does selective context propagation via an attentive fusion module enhance relational reasoning in HOI detection?
• RQ3 How does a three-branch architecture compare to two-branch or single-branch designs in terms of feature discrimination and performance?
• RQ4 To what extent does disentangling human and object branches improve detection accuracy compared to parameter sharing?
• RQ5 Can rich context exchange between branches mitigate the limitations of disentangled transformers in relational reasoning?

Key findings

• MUREN achieves state-of-the-art performance on HICO-DET with an AP of 68.8 and on V-COCO with an AP of 71.0, outperforming prior methods.
• Ablation studies show that removing the attentive fusion module reduces performance by 2.25%p on HICO-DET and 2.04%p on V-COCO, proving its effectiveness.
• The combination of 'conditioning' and 'channel attention' in the fusion module yields the best performance, confirming that sub-tasks require distinct context.
• Sharing parameters between human and object branches significantly degrades performance—MUREN-(6) drops by 2.2%p on HICO-DET and 1.9%p on V-COCO compared to the non-shared MUREN-(0).
• Visualizations confirm that the interaction branch attends to regions where interactions occur, and the MURE module highlights holistic relational regions, validating its role in relational reasoning.
• The model with full disentanglement (MUREN-(0)) achieves higher accuracy than a parameter-matched variant with shared layers (MUREN†), proving the benefit of dedicated human and object encoders.

Figure 2 : The overall architecture of MUREN. The proposed method adopts three-branch architecture: human branch, object branch, and interaction branch. Each branch is responsible for human detection, object detection, interaction classification. The input image is fed into the CNN backbone followed