[Paper Review] TMHOI: Translational Model for Human-Object Interaction Detection

Lijing Zhu, Qizhen Lan+5 more|arXiv (Cornell University)|Mar 7, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes TMHOI, a graph-based model for human-object interaction (HOI) detection that integrates spatial and semantic knowledge via a translational knowledge graph embedding model. By encoding both appearance and relational features into node embeddings using a 50-dimensional embedding space, TMHOI achieves state-of-the-art performance on HICO-DET with 26.95 mAP, outperforming prior graph-based methods while maintaining computational efficiency.

ABSTRACT

Detecting human-object interactions (HOIs) is an intricate challenge in the field of computer vision. Existing methods for HOI detection heavily rely on appearance-based features, but these may not fully capture all the essential characteristics necessary for accurate detection. To overcome these challenges, we propose an innovative graph-based approach called TMGHOI (Translational Model for Human-Object Interaction Detection). Our method effectively captures the sentiment representation of HOIs by integrating both spatial and semantic knowledge. By representing HOIs as a graph, where the interaction components serve as nodes and their spatial relationships as edges. To extract crucial spatial and semantic information, TMGHOI employs separate spatial and semantic encoders. Subsequently, these encodings are combined to construct a knowledge graph that effectively captures the sentiment representation of HOIs. Additionally, the ability to incorporate prior knowledge enhances the understanding of interactions, further boosting detection accuracy. We conducted extensive evaluations on the widely-used HICO-DET datasets to demonstrate the effectiveness of TMGHOI. Our approach outperformed existing state-of-the-art graph-based methods by a significant margin, showcasing its potential as a superior solution for HOI detection. We are confident that TMGHOI has the potential to significantly improve the accuracy and efficiency of HOI detection. Its integration of spatial and semantic knowledge, along with its computational efficiency and practicality, makes it a valuable tool for researchers and practitioners in the computer vision community. As with any research, we acknowledge the importance of further exploration and evaluation on various datasets to establish the generalizability and robustness of our proposed method.

Motivation & Objective

• Address the limitations of appearance-only features in existing HOI detection methods, which fail to capture relational and contextual knowledge.
• Overcome the high computational cost of transformer-based models in HOI detection by introducing a more efficient graph-based alternative.
• Improve detection accuracy by integrating prior knowledge through knowledge graph embeddings that model spatial and semantic relationships.
• Enable practical deployment in real-world applications by balancing model expressiveness and computational efficiency.
• Demonstrate the effectiveness of translational knowledge graph models in enhancing node representations for HOI detection tasks.

Proposed method

• Construct a knowledge graph where human and object instances are nodes, and spatial relationships are edges.
• Use a spatial encoder to extract spatial layout features between human-object pairs and a semantic encoder for visual and contextual semantics.
• Employ a translational model (e.g., TransE-style) to embed relations into a low-dimensional vector space, with relation-specific hyperplanes.
• Combine appearance features with relation embeddings via element-wise concatenation or residual connection to form enriched node representations.
• Train the model end-to-end using multi-label classification loss on the HICO-DET dataset, optimizing for mAP.
• Conduct ablation studies to determine optimal embedding size (k=50) balancing expressiveness and overfitting.

Experimental results

Research questions

• RQ1 Can a translational knowledge graph embedding model effectively capture relational knowledge in human-object interaction detection?
• RQ2 How does integrating spatial and semantic features through graph-based encoding improve HOI detection performance compared to appearance-only baselines?
• RQ3 What is the optimal embedding dimension for the translational model in this HOI detection setting?
• RQ4 Does the proposed method achieve better accuracy and efficiency than existing graph-based and transformer-based HOI detectors?
• RQ5 Can the model generalize well to rare and non-rare interaction types, particularly when incorporating relational inductive bias?

Key findings

• TMHOI achieves a mean average precision (mAP) of 26.95 on the HICO-DET dataset, outperforming prior state-of-the-art graph-based methods.
• The ablation study confirms that an embedding size of k=50 yields the best performance, balancing model capacity and generalization.
• Models without the translational relation embedding perform significantly worse, with an mAP drop of nearly 1 point, highlighting the importance of relational features.
• The model demonstrates improved consistency between node and edge embeddings due to the joint optimization of spatial, semantic, and relational representations.
• The method maintains computational efficiency compared to transformer-based models, making it suitable for real-world deployment.
• The model achieves strong performance on both rare (21.28 mAP) and non-rare (28.56 mAP) interaction categories, indicating robustness across interaction types.