[论文解读] TMHOI: Translational Model for Human-Object Interaction Detection

Lijing Zhu, Qizhen Lan+5 more|arXiv (Cornell University)|Mar 7, 2023

Multimodal Machine Learning Applications被引用 4

一句话总结

本文提出TMHOI，一种基于图的模型，用于人体-物体交互（HOI）检测，通过一种平移知识图嵌入模型整合空间与语义知识。通过在50维嵌入空间中编码外观特征与关系特征，TMHOI在HICO-DET数据集上实现了26.95 mAP的最先进性能，优于以往基于图的方法，同时保持了计算效率。

ABSTRACT

Detecting human-object interactions (HOIs) is an intricate challenge in the field of computer vision. Existing methods for HOI detection heavily rely on appearance-based features, but these may not fully capture all the essential characteristics necessary for accurate detection. To overcome these challenges, we propose an innovative graph-based approach called TMGHOI (Translational Model for Human-Object Interaction Detection). Our method effectively captures the sentiment representation of HOIs by integrating both spatial and semantic knowledge. By representing HOIs as a graph, where the interaction components serve as nodes and their spatial relationships as edges. To extract crucial spatial and semantic information, TMGHOI employs separate spatial and semantic encoders. Subsequently, these encodings are combined to construct a knowledge graph that effectively captures the sentiment representation of HOIs. Additionally, the ability to incorporate prior knowledge enhances the understanding of interactions, further boosting detection accuracy. We conducted extensive evaluations on the widely-used HICO-DET datasets to demonstrate the effectiveness of TMGHOI. Our approach outperformed existing state-of-the-art graph-based methods by a significant margin, showcasing its potential as a superior solution for HOI detection. We are confident that TMGHOI has the potential to significantly improve the accuracy and efficiency of HOI detection. Its integration of spatial and semantic knowledge, along with its computational efficiency and practicality, makes it a valuable tool for researchers and practitioners in the computer vision community. As with any research, we acknowledge the importance of further exploration and evaluation on various datasets to establish the generalizability and robustness of our proposed method.

研究动机与目标

• 解决现有HOI检测方法中仅依赖外观特征的局限性，这些方法无法捕捉关系与上下文知识。
• 通过引入更高效的基于图的替代方案，克服基于Transformer模型在HOI检测中的高计算成本。
• 通过建模空间与语义关系的知识图嵌入，整合先验知识以提升检测准确率。
• 通过平衡模型表达能力与计算效率，实现实际应用场景中的可部署性。
• 证明平移知识图模型在增强HOI检测任务中节点表示方面的有效性。

提出的方法

• 构建一个知识图，其中人体与物体实例为节点，空间关系为边。
• 使用空间编码器提取人体-物体对之间的空间布局特征，使用语义编码器提取视觉与上下文语义信息。
• 采用平移模型（如TransE风格）将关系嵌入低维向量空间，并引入关系特定的超平面。
• 通过逐元素拼接或残差连接，将外观特征与关系嵌入结合，形成增强的节点表示。
• 在HICO-DET数据集上使用多标签分类损失端到端训练模型，以mAP为优化目标。
• 通过消融研究确定最优嵌入维度（k=50），在表达能力与过拟合之间取得平衡。

实验结果

研究问题

• RQ1 平移知识图嵌入模型能否有效捕捉人体-物体交互检测中的关系知识？
• RQ2 通过基于图的编码整合空间与语义特征，相较于仅依赖外观的基线方法，能否显著提升HOI检测性能？
• RQ3 在本HOI检测设置下，平移模型的最优嵌入维度是多少？
• RQ4 所提方法是否在准确率与效率方面优于现有的基于图与基于Transformer的HOI检测器？
• RQ5 当引入关系归纳偏置时，模型能否在罕见与非罕见交互类型上均表现出良好的泛化能力？

主要发现

• TMHOI在HICO-DET数据集上实现了26.95 mAP的平均平均精度，优于以往最先进基于图的方法。
• 消融研究证实，嵌入维度k=50时性能最佳，兼顾了模型容量与泛化能力。
• 缺乏平移关系嵌入的模型性能显著下降，mAP下降近1个百分点，凸显了关系特征的重要性。
• 由于空间、语义与关系表示的联合优化，模型在节点与边嵌入之间表现出更高的连贯性。
• 与基于Transformer的模型相比，该方法保持了计算效率，适合实际部署。
• 模型在罕见交互类型（21.28 mAP）与非罕见交互类型（28.56 mAP）上均表现优异，表明其在各类交互类型上的鲁棒性。