[Paper Review] Category Query Learning for Human-Object Interaction Classification

Chi Xie, Fangao Zeng+3 more|arXiv (Cornell University)|Mar 24, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Category Query Learning (CQL), a novel method for human-object interaction (HOI) classification that replaces static category weight vectors with learnable, category-specific queries. These queries are optimized via an image-level classification head and adaptively represent interaction categories, achieving state-of-the-art performance on HICO-DET (36.03 mAP) across multiple baselines with minimal computational overhead.

ABSTRACT

Unlike most previous HOI methods that focus on learning better human-object features, we propose a novel and complementary approach called category query learning. Such queries are explicitly associated to interaction categories, converted to image specific category representation via a transformer decoder, and learnt via an auxiliary image-level classification task. This idea is motivated by an earlier multi-label image classification method, but is for the first time applied for the challenging human-object interaction classification task. Our method is simple, general and effective. It is validated on three representative HOI baselines and achieves new state-of-the-art results on two benchmarks.

Motivation & Objective

• To address the limitation of static category weight vectors in HOI classification, which fail to model complex appearance variations across interaction categories.
• To improve interaction classification by learning dynamic, image-adaptive representations for each interaction category.
• To introduce a simple, general, and effective method that complements existing HOI frameworks without major architectural changes.
• To validate the effectiveness of image-level category query learning, especially on dense human-object interaction scenes.

Proposed method

• Introduces category queries that are explicitly associated with specific interaction categories and trained via an auxiliary image-level classification task.
• Uses a transformer decoder to convert category queries into image-specific category representations, enabling adaptive modeling of interaction semantics.
• Replaces the traditional linear classifier's static weight vectors with these learnable queries, maintaining the same inference mechanism.
• The method is lightweight and general, compatible with any off-the-shelf HOI model that outputs human-object features.
• The queries are category-specific and distinct from instance-level queries used in other HOI transformers, focusing on global image-level semantics.
• Training involves joint optimization of HOI detection and image-level category classification, stabilizing and guiding category representation learning.

Experimental results

Research questions

• RQ1 Can learnable, category-specific queries improve interaction classification performance in HOI detection compared to static weight vectors?
• RQ2 Does image-level category query learning provide better generalization, especially on complex or dense interaction scenes?
• RQ3 How does the proposed method compare in performance and robustness to existing HOI baselines across different benchmarks?
• RQ4 Is the improvement of the proposed method more pronounced in images with higher human-object interaction density?
• RQ5 Can the method be effectively integrated into diverse HOI frameworks without architectural overhaul?

Key findings

• The proposed method achieves a new state-of-the-art mAP of 36.03 on the HICO-DET benchmark, outperforming all three tested baselines.
• The method consistently improves all three baseline models—two transformer-based and one two-stage—across all evaluation settings.
• The relative improvement of the method increases with interaction density, indicating stronger gains on challenging, dense scenes.
• Qualitative analysis shows that the model suppresses false positives (e.g., FP "board" reduced from 0.29 to 0.07) and boosts true positives (e.g., TP "sit on bus" increased from 0.15 to 0.38).
• The method enables correct image-level category scoring, where incorrect categories receive low confidence scores, enhancing overall detection reliability.