[Paper Review] UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection

Bumsoo Kim, Tae-Ho Choi+2 more|arXiv (Cornell University)|Dec 19, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes UnionDet, a one-stage human-object interaction (HOI) detection framework that directly predicts interaction regions (unions of human-object pairs) using a novel union-level detector, eliminating the need for post-object-detection pairing. By introducing union anchor labeling, target object classification loss, and foreground focal loss, UnionDet achieves 4×–14× faster inference for interaction prediction while outperforming state-of-the-art methods on V-COCO and HICO-DET.

ABSTRACT

Recent advances in deep neural networks have achieved significant progress in detecting individual objects from an image. However, object detection is not sufficient to fully understand a visual scene. Towards a deeper visual understanding, the interactions between objects, especially humans and objects are essential. Most prior works have obtained this information with a bottom-up approach, where the objects are first detected and the interactions are predicted sequentially by pairing the objects. This is a major bottleneck in HOI detection inference time. To tackle this problem, we propose UnionDet, a one-stage meta-architecture for HOI detection powered by a novel union-level detector that eliminates this additional inference stage by directly capturing the region of interaction. Our one-stage detector for human-object interaction shows a significant reduction in interaction prediction time 4x~14x while outperforming state-of-the-art methods on two public datasets: V-COCO and HICO-DET.

Motivation & Objective

• Address the inefficiency of multi-stage HOI detectors that require sequential object detection and interaction pairing, which creates a major inference bottleneck.
• Enable real-time HOI detection by eliminating the need for heavy post-processing inference on object pairs.
• Develop a unified, end-to-end trainable one-stage architecture that detects objects and interactions simultaneously.
• Overcome the technical challenges of union-level detection, such as bias toward human regions and IoU-based matching inaccuracies.
• Achieve state-of-the-art performance with minimal additional inference time beyond standard object detection.

Proposed method

• Propose a novel union-level detector that directly regresses the interaction region (union of human and object) instead of relying on post-hoc pairing of detected objects.
• Introduce union anchor labeling to assign anchors to interaction regions based on union-level IoU matching, improving localization accuracy.
• Design a target object classification loss ($\mathcal{L}^{cls}_{ij}$) to better distinguish the object class in interaction pairs.
• Develop a union foreground focal loss (FFL) that reweights positive samples based on union-level confidence, reducing background bias.
• Use a union matching function ($\mu_u$) that replaces standard IoU with a more robust metric for matching anchors to interaction regions.
• Integrate the union detector into a meta-architecture (UnionDet) compatible with existing one-stage object detectors, enabling end-to-end training.

Figure 1: Technical Challenges in Union Detection. (Left) The box with the highest confidence for the ground-truth action ’throw’ is highlighted in bold, and the region with the highest IoU with the ground-truth union region is dotted. As you can see, the highest confidence is biased towards the hum

Experimental results

Research questions

• RQ1 Can a one-stage detector directly predict human-object interaction regions without relying on a separate pairing stage?
• RQ2 How can union-level detection overcome challenges such as bias toward human regions and inaccurate IoU-based matching?
• RQ3 What training techniques are necessary to stabilize and improve the performance of union-level detection?
• RQ4 To what extent can a union-level detector reduce inference time compared to multi-stage HOI detectors?
• RQ5 Can a union-level detector achieve state-of-the-art performance while maintaining real-time inference speed?

Key findings

• UnionDet achieves 4×–14× faster inference for interaction prediction compared to baseline multi-stage HOI detectors, which require 40ms–130ms per image after object detection.
• The model reduces additional inference time to only 9.06ms (11.7%) on top of a standard object detector, significantly outperforming previous methods.
• On the V-COCO dataset, UnionDet achieves 47.5% and 56.2% average precision in two evaluation scenarios, surpassing state-of-the-art methods.
• On HICO-DET, UnionDet achieves state-of-the-art performance in both the 'Default' and 'Known Object' settings without relying on external features.
• Ablation studies confirm that each component—union branch, target object classification loss, union matching function, and foreground focal loss—significantly improves detection performance.
• The model maintains high performance while being fully end-to-end trainable and compatible with existing one-stage object detectors, enabling real-time deployment.

Figure 2: The overall architecture of UnionDet. Our UnionDet is generally compatible with one-stage object detectors. The feature pyramid obtained from the backbone network is simultaneously fed to Union Branch and Instance Branch. While Union Branch directly captures the region of interaction, Inst