[Paper Review] QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information

Masato Tamura, Hiroki Ohashi+1 more|arXiv (Cornell University)|Mar 9, 2021

Advanced Neural Network ApplicationsComputer Science50 references17 citations

TL;DR

QPIC proposes a query-based, transformer-powered human-object interaction (HOI) detection method that leverages image-wide contextual features through self-attention and dedicated queries per HOI pair, avoiding feature contamination in overlapping instances. It achieves state-of-the-art performance, improving mAP by 5.37 on HICO-DET and 5.7 on V-COCO.

ABSTRACT

We propose a simple, intuitive yet powerful method for human-object interaction (HOI) detection. HOIs are so diverse in spatial distribution in an image that existing CNN-based methods face the following three major drawbacks; they cannot leverage image-wide features due to CNN's locality, they rely on a manually defined location-of-interest for the feature aggregation, which sometimes does not cover contextually important regions, and they cannot help but mix up the features for multiple HOI instances if they are located closely. To overcome these drawbacks, we propose a transformer-based feature extractor, in which an attention mechanism and query-based detection play key roles. The attention mechanism is effective in aggregating contextually important information image-wide, while the queries, which we design in such a way that each query captures at most one human-object pair, can avoid mixing up the features from multiple instances. This transformer-based feature extractor produces so effective embeddings that the subsequent detection heads may be fairly simple and intuitive. The extensive analysis reveals that the proposed method successfully extracts contextually important features, and thus outperforms existing methods by large margins (5.37 mAP on HICO-DET, and 5.7 mAP on V-COCO). The source codes are available at $\href{https://github.com/hitachi-rd-cv/qpic}{ ext{this https URL}}$.

Motivation & Objective

• Address the limitations of CNN-based HOI detectors that fail to utilize global context due to local receptive fields.
• Overcome feature contamination in overlapping HOI instances by decoupling feature extraction per interaction pair.
• Improve detection robustness in scenarios with distant humans/objects or missing contextual cues outside bounding boxes.
• Design a simple, effective detection head by leveraging powerful, context-aware features from a transformer-based architecture.
• Demonstrate that the transformer decoder is essential for HOI detection, especially for globally distributed interactions.

Proposed method

• Employ a vision transformer encoder to extract image-wide features using self-attention, enabling selective aggregation of contextually relevant regions.
• Design instance-specific queries such that each query attends to at most one human-object pair, preventing feature mixing across multiple interactions.
• Use a transformer decoder with learned queries to refine features and generate detection heads for human, object, and action prediction.
• Integrate the query-based detection head with a point-matching mechanism for efficient and accurate interaction prediction.
• Utilize a ResNet-101 backbone for feature extraction, followed by the transformer encoder-decoder stack for contextual feature learning.
• Train end-to-end with cross-entropy and box regression losses for human, object, and action classification, and bounding box regression.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture with image-wide attention improve HOI detection by capturing context beyond human-object bounding boxes?
• RQ2 Does query-based detection with instance-specific queries prevent feature contamination in overlapping HOI instances?
• RQ3 How effective is the transformer decoder in capturing globally distributed interactions compared to standard CNN-based feature extractors?
• RQ4 To what extent does the proposed method outperform existing two-stage and single-stage HOI detectors on standard benchmarks?
• RQ5 What role does the attention mechanism play in detecting interactions when key contextual cues lie outside the human-object bounding boxes?

Key findings

• QPIC achieves 29.27 mAP on HICO-DET and 43.5 mAP on COCO, outperforming prior state-of-the-art methods by 5.37 mAP and 5.7 mAP, respectively.
• The transformer decoder contributes significantly to performance, improving mAP by 9.20 (with encoder) and 7.86 (without encoder) over the encoder-only variant.
• The model with both encoder and decoder achieves 45.8% relative improvement in HOI detection on HICO-DET, compared to 23.9% on object detection, indicating higher sensitivity to global context.
• Qualitative analysis confirms that QPIC’s attention maps selectively focus on relevant contextual cues—such as a hose in a distant interaction—while ignoring distractors like irrelevant humans.
• QPIC successfully detects interactions in challenging cases such as distant humans/objects (e.g., midpoint-based methods fail), and overlapping instances where feature contamination is common.
• The ablation study shows that the combination of image-wide attention and query-based detection is essential, as removing either component leads to significant performance drops.