[Paper Review] HOI Analysis: Integrating and Decomposing Human-Object Interaction

Yong–Lu Li, Xinpeng Liu+3 more|arXiv (Cornell University)|Oct 30, 2020

Multimodal Machine Learning ApplicationsComputer Science52 references64 citations

TL;DR

The paper proposes HOI Analysis with an Integration-Decomposition Network (IDN) that represents verbs in transformation function space by decomposing and integrating human and object features, achieving state-of-the-art HOI detection on benchmarks.

ABSTRACT

Human-Object Interaction (HOI) consists of human, object and implicit interaction/verb. Different from previous methods that directly map pixels to HOI semantics, we propose a novel perspective for HOI learning in an analytical manner. In analogy to Harmonic Analysis, whose goal is to study how to represent the signals with the superposition of basic waves, we propose the HOI Analysis. We argue that coherent HOI can be decomposed into isolated human and object. Meanwhile, isolated human and object can also be integrated into coherent HOI again. Moreover, transformations between human-object pairs with the same HOI can also be easier approached with integration and decomposition. As a result, the implicit verb will be represented in the transformation function space. In light of this, we propose an Integration-Decomposition Network (IDN) to implement the above transformations and achieve state-of-the-art performance on widely-used HOI detection benchmarks. Code is available at https://github.com/DirtyHarryLYL/HAKE-Action-Torch/tree/IDN-(Integrating-Decomposing-Network).

Motivation & Objective

• Motivate a new perspective on HOI by analyzing inner structure of HOI beyond pixel-to-semantics mappings.
• Propose a latent-space transformation framework that integrates and decomposes human and object features to encode verbs.
• Learn verbs in a transformation function space to improve inter-pair transformations and rare HOI handling.
• Demonstrate state-of-the-art HOI detection performance on standard benchmarks and show robustness to detector variations.

Proposed method

• Represent HOI as a transformation between isolated human/object features and a coherent HOI embedding.
• Use an auto-encoder to compress features and learn verb-specific integration T_I(·) and decomposition T_D(·) mappings.
• Model inter-pair transformations by exchanging similar human/object instances within the same HOI to learn verb space without motion transfer.
• Train with objectives enforcing integration validity, decomposition validity, and interactiveness validity in the transformation space.
• Apply integration and decomposition to detect HOI by comparing distances to verb-specific integrated features and combining multiple verb signals.
• Operate all transformations in latent space to enable efficient, parallelizable learning for multiple verbs.

Experimental results

Research questions

• RQ1 How can HOI be represented as a structured transformation between isolated elements and the whole interaction?
• RQ2 Can integration and decomposition in latent space capture verb semantics better than direct pixel-to-verb mappings?
• RQ3 Does exchanging similar instances across HOI pairs uncover shared verb representations and improve learning for rare HOIs?
• RQ4 Can verb representation be effectively learned in transformation function space and used for robust HOI detection?

Key findings

• IDN achieves state-of-the-art results on HICO-DET with a COCO detector, reaching 23.36 mAP on the Default Full set.
• IDN improves performance substantially with a detector finetuned on HICO-DET, reaching 26.29 mAP on Default Full.
• On GT boxes, IDN attains a large margin over prior methods, with 43.98 mAP for Full on HICO-DET.
• IDN attains 53.3 AP_role on V-COCO Scenario 1 and 60.3 AP_role on Scenario 2, outperforming prior methods.
• Ablation studies show that integrating, decomposing, IPT, and AE components each contribute to performance gains.
• The method operates at 10.04 FPS on a single GPU, demonstrating good efficiency and scalability.