[Paper Review] Improving Human-Object Interaction Detection via Phrase Learning and Label Composition

Zhimin Li, Cheng Zou+3 more|arXiv (Cornell University)|Dec 14, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes PhraseHOI, a multi-task framework that improves Human-Object Interaction (HOI) detection by jointly training a HOI branch and a novel relational phrase learning branch using language priors. By leveraging word2vec embeddings and a label composition method based on semantic neighbors, PhraseHOI addresses the long-tailed distribution in HOI and achieves state-of-the-art performance on HICO-DET, surpassing prior methods on Full and NonRare splits with a 29.29% AP on Full and 31.46% on NonRare.

ABSTRACT

Human-Object Interaction (HOI) detection is a fundamental task in high-level human-centric scene understanding. We propose PhraseHOI, containing a HOI branch and a novel phrase branch, to leverage language prior and improve relation expression. Specifically, the phrase branch is supervised by semantic embeddings, whose ground truths are automatically converted from the original HOI annotations without extra human efforts. Meanwhile, a novel label composition method is proposed to deal with the long-tailed problem in HOI, which composites novel phrase labels by semantic neighbors. Further, to optimize the phrase branch, a loss composed of a distilling loss and a balanced triplet loss is proposed. Extensive experiments are conducted to prove the effectiveness of the proposed PhraseHOI, which achieves significant improvement over the baseline and surpasses previous state-of-the-art methods on Full and NonRare on the challenging HICO-DET benchmark.

Motivation & Objective

• To improve human-object interaction detection by leveraging language priors through a novel relational phrase learning task.
• To address the long-tailed distribution problem in HOI by composing novel labels using semantic neighbors in the label space.
• To jointly optimize HOI detection and phrase learning via a multi-task architecture, enhancing feature discriminability and generalization.
• To demonstrate that knowledge distillation from language embeddings improves HOI performance without requiring additional annotations.

Proposed method

• Introduces a relational phrase learning task that generates natural language phrases describing human-object interactions, with ground truths automatically derived from existing HOI annotations.
• Proposes a multi-task network, PhraseHOI, with a shared transformer backbone and two parallel branches: one for HOI detection and one for phrase embedding prediction.
• Uses a Look-Up-Table (LUT) to decode learned phrase embeddings into natural language phrases.
• Employs a distilling loss to align predicted phrase embeddings with pre-trained word2vec embeddings, preserving global semantic structure.
• Applies a balanced triplet loss to enhance local discriminability between semantically similar but different phrases.
• Develops a label composition method that generates new HOI labels by replacing verbs or objects with their semantic neighbors, effectively expanding the label space for rare categories.

Experimental results

Research questions

• RQ1 Can relational phrase learning, derived from existing HOI annotations, improve the performance of HOI detection through multi-task learning?
• RQ2 How can language priors from pre-trained word embeddings be effectively distilled into the HOI detection pipeline?
• RQ3 Can label composition based on semantic neighbors mitigate the long-tailed distribution problem in HOI datasets?
• RQ4 Does joint optimization of HOI and phrase learning lead to better generalization and zero-shot capability?

Key findings

• PhraseHOI achieves a 29.29% average precision (AP) on the Full split and 31.46% on the NonRare split of the HICO-DET benchmark, setting a new state-of-the-art.
• The ablation study shows that using the phrase branch with both distilling and triplet losses leads to a 1.19% AP gain on Full and a 3.39% gain on Rare, demonstrating the effectiveness of the loss design.
• The concatenated embedding method outperforms language model-encoded embeddings, with a 29.29% AP vs. 22.64% AP, due to better discriminative ability in the embedding space.
• Replacing word2vec embeddings with random embeddings reduces performance to 21.90% AP, confirming that language prior knowledge, not just continuous encoding, is key to the improvement.
• The label composition method effectively boosts rare category performance, with a 3.39% AP gain on Rare when using m=0.50 in the triplet loss.
• The phrase branch enables knowledge distillation from language space, improving the HOI branch even when the phrase branch is not used at inference time.