[论文解读] Diagnosing Human-object Interaction Detectors

Fangrui Zhu, Yiming Xie+2 more|arXiv (Cornell University)|Aug 16, 2023

Multimodal Machine Learning ApplicationsComputer Science被引用 3

一句话总结

本文提出了一种用于人体-物体交互（HOI）检测的诊断工具箱，提供超越mAP的详细、定量的模型性能分析。通过将错误分解为可检测的类型，并测量从Oracle修复中获得的mAP提升，作者识别出关键的失败模式——尤其在交互分类方面——并揭示当前最先进模型RLIPv2之所以表现更优，主要归因于其在多标签交互分类准确性方面的显著优势。

ABSTRACT

We have witnessed significant progress in human-object interaction (HOI) detection. The reliance on mAP (mean Average Precision) scores as a summary metric, however, does not provide sufficient insight into the nuances of model performance (e.g., why one model is better than another), which can hinder further innovation in this field. To address this issue, in this paper, we introduce a diagnosis toolbox to provide detailed quantitative break-down analysis of HOI detection models, inspired by the success of object detection diagnosis toolboxes. We first conduct holistic investigations in the pipeline of HOI detection. By defining a set of errors and the oracles to fix each of them, we can have a quantitative analysis of the significance of different errors according to the mAP improvement obtained from fixing each error. We then delve into two sub-tasks of HOI detection: human-object pair detection and interaction classification, respectively. For the first detection task, we compute the coverage of ground-truth human-object pairs as well as the noisiness level in the detection results. For the second classification task, we measure a model's performance of differentiating positive and negative detection results and also classifying the actual interactions when the human-object pairs are correctly detected. We analyze eight state-of-the-art HOI detection models and provide valuable diagnosis insights to foster future research. For instance, our diagnosis shows that state-of-the-art model RLIPv2 outperforms others mainly because it significantly improves the multi-label interaction classification accuracy. Our toolbox is applicable for different methods across different datasets and available at https://github.com/neu-vi/Diag-HOI.

研究动机与目标

• 解决尽管mAP得分持续提升，但对HOI检测模型性能缺乏深入洞察的问题。
• 通过量化不同错误类型对mAP的影响，识别模型失败的根本原因。
• 将HOI检测分解为两个子任务——人体-物体对检测与交互分类，以实现精细化分析。
• 提供一种适用于不同模型与数据集的诊断框架，以指导未来模型开发。
• 通过详细定量分解，揭示最先进模型如RLIPv2为何表现更优。

提出的方法

• 在HOI检测流程中定义一组错误类型（如误报、漏检），并引入Oracle以修复每类错误。
• 通过测量应用每类Oracle后mAP的提升，量化各类错误的重要性。
• 通过配对召回率（对真实样本对的覆盖程度）和配对精确率（检测的噪声程度）评估人体-物体对检测。
• 通过AP（用于正/负对的二分类）和mAP（当配对正确时的多标签分类）评估交互分类。
• 将该工具箱应用于八个SOTA HOI模型在多个数据集上的表现，分离子任务间的性能表现。
• 通过NMS和检测阈值调整，比较单阶段与两阶段模型在配对数量相当条件下的表现。

实验结果

研究问题

• RQ1 在HOI检测模型中，哪些类型的错误对mAP下降的贡献最为显著？
• RQ2 单阶段与两阶段HOI检测范式在配对检测与分类性能方面如何比较？
• RQ3 当前HOI检测的主要瓶颈是配对定位还是交互分类？
• RQ4 为何最先进模型RLIPv2尽管整体mAP相近，仍优于其他模型？
• RQ5 物体检测质量与人体检测召回率在多大程度上影响最终HOI性能？

主要发现

• 修复误报错误可带来显著的mAP提升，部分模型在纠正此类错误后mAP提升超过30%。
• 漏检错误对性能损失的贡献显著，当修复缺失的真实样本对时，mAP提升超过30%。
• RLIPv2优于其他模型，主要归因于修复交互分类错误带来的12.94% mAP提升，表明其在多标签分类方面具有显著优势。
• 配对检测的召回率通常较高（如在HICO-DET上人体检测的召回率为91.5%），但精确率较低，表明检测结果存在噪声。
• 提高检测到的配对数量（通过降低阈值或禁用NMS）可提升召回率，但对最终mAP的提升不显著，表明配对质量比数量更为关键。
• 诊断结果表明，错误的交互分类是主要瓶颈，尤其在依赖弱监督或端到端训练的模型中更为明显。