[Paper Review] Mining the Benefits of Two-stage and One-stage HOI Detection

Aixi Zhang, Yue Liao+5 more|arXiv (Cornell University)|Aug 11, 2021

Multimodal Machine Learning Applications38 references35 citations

TL;DR

The paper analyzes two-stage and one-stage HOI detectors, and proposes Cascade Disentangling Network (CDN), a one-stage framework that separates human-object pair detection from interaction classification, yielding large gains on HICO-Det especially for rare classes.

ABSTRACT

Two-stage methods have dominated Human-Object Interaction (HOI) detection for several years. Recently, one-stage HOI detection methods have become popular. In this paper, we aim to explore the essential pros and cons of two-stage and one-stage methods. With this as the goal, we find that conventional two-stage methods mainly suffer from positioning positive interactive human-object pairs, while one-stage methods are challenging to make an appropriate trade-off on multi-task learning, i.e., object detection, and interaction classification. Therefore, a core problem is how to take the essence and discard the dregs from the conventional two types of methods. To this end, we propose a novel one-stage framework with disentangling human-object detection and interaction classification in a cascade manner. In detail, we first design a human-object pair generator based on a state-of-the-art one-stage HOI detector by removing the interaction classification module or head and then design a relatively isolated interaction classifier to classify each human-object pair. Two cascade decoders in our proposed framework can focus on one specific task, detection or interaction classification. In terms of the specific implementation, we adopt a transformer-based HOI detector as our base model. The newly introduced disentangling paradigm outperforms existing methods by a large margin, with a significant relative mAP gain of 9.32% on HICO-Det. The source codes are available at https://github.com/YueLiao/CDN.

Motivation & Objective

• Assess the strengths and weaknesses of conventional two-stage and one-stage HOI detectors.
• Propose a unified, end-to-end one-stage framework that disentangles detection and interaction classification.
• Show that cascade disentangling improves HOI detection performance, especially for rare classes.

Proposed method

• Introduce Cascade Disentangling Network (CDN) with two cascade decoders: Human-Object Pair Decoder (HO-PD) and Interaction Decoder.
• HO-PD removes the interaction head to focus on detecting human-object pairs, producing HO predictions and an interactive score.
• Use HO-PD outputs as queries to initialize the Interaction Decoder, which predicts action categories for each HO pair.
• Adopt a transformer-based detector as the base (inheriting from DETR/previous HOI work) and train with a bipartite matching loss similar to QPIC.
• Apply a decoupling dynamic re-weighting scheme to mitigate long-tailed HOI distributions during training.
• During inference, employ Pair-wise Non-Maximal Suppression (PNMS) to refine HOI triplets.

Experimental results

Research questions

• RQ1 Can disentangling detection and interaction classification in a cascade within a one-stage framework outperform traditional two-stage HOI detectors?
• RQ2 Does a cascade, as opposed to parallel multi-task heads, improve feature specialization for detection versus interaction classification?
• RQ3 How does the proposed CDN perform on standard HOI benchmarks (HICO-Det, V-COCO) compared to state-of-the-art methods, particularly for rare classes?
• RQ4 What training strategies (e.g., dynamic re-weighting, PNMS) contribute to alleviating long-tail issues in HOI detection?

Key findings

• CDN significantly outperforms prior methods on HOI detection, achieving a large margin gain on HICO-Det (e.g., CDN-B and CDN-L achieve top results).
• CDN-S base outperforms QPIC by 6.50% in full mAP and shows particularly strong improvement for rare classes (e.g., +23.66% relative on rare HOIs).
• The disentangled cascade (HO-PD followed by isolated interaction decoding) yields better specialization: HO-PD focuses on detection regions, while the Interaction Decoder emphasizes interaction context (e.g., hands for holding).
• On HICO-Det, CDN-S/B/L achieve 31.44/31.78/32.07 mAP, respectively, versus 29.07 for QPIC, marking substantial gains; on V-COCO, CDN-B achieves 64.42 AP_role (Scenario 2) and 62.29 (Scenario 1).
• The dynamic re-weighting and PNMS contribute additional performance gains, with PNMS pushing CDN-B to 31.78 mAP on HICO-Det.