[Paper Review] ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection

Jeeseung Park, Jinwoo Park+1 more|arXiv (Cornell University)|Apr 17, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

ViPLO proposes a Vision Transformer-based two-stage human-object interaction (HOI) detector that improves performance through a novel Masking with Overlapped Area (MOA) module for quantization-free feature extraction and a pose-conditioned self-loop graph for joint-aware interaction classification. It achieves state-of-the-art performance, gaining +2.07 mAP on HICO-DET and outperforming prior methods while maintaining efficiency.

ABSTRACT

Human-Object Interaction (HOI) detection, which localizes and infers relationships between human and objects, plays an important role in scene understanding. Although two-stage HOI detectors have advantages of high efficiency in training and inference, they suffer from lower performance than one-stage methods due to the old backbone networks and the lack of considerations for the HOI perception process of humans in the interaction classifiers. In this paper, we propose Vision Transformer based Pose-Conditioned Self-Loop Graph (ViPLO) to resolve these problems. First, we propose a novel feature extraction method suitable for the Vision Transformer backbone, called masking with overlapped area (MOA) module. The MOA module utilizes the overlapped area between each patch and the given region in the attention function, which addresses the quantization problem when using the Vision Transformer backbone. In addition, we design a graph with a pose-conditioned self-loop structure, which updates the human node encoding with local features of human joints. This allows the classifier to focus on specific human joints to effectively identify the type of interaction, which is motivated by the human perception process for HOI. As a result, ViPLO achieves the state-of-the-art results on two public benchmarks, especially obtaining a +2.07 mAP performance gain on the HICO-DET dataset. The source codes are available at https://github.com/Jeeseung-Park/ViPLO.

Motivation & Objective

• To improve the performance of two-stage HOI detectors, which have been outperformed by one-stage methods despite their efficiency advantages.
• To address the quantization problem when using Vision Transformers (ViT) with region-based feature extraction in HOI detection.
• To model human perception of HOI by incorporating pose-aware, joint-focused reasoning into the interaction prediction process.
• To maintain high inference speed and memory efficiency while leveraging the representational power of ViT backbones.

Proposed method

• Introduces the Masking with Overlapped Area (MOA) module, which computes attention using the overlapping area between ViT patches and object regions to avoid spatial quantization.
• Uses ViT as the backbone network with MOA to extract high-fidelity features for human and object regions without relying on ROIAlign with quantized patch indexing.
• Designs a pose-conditioned self-loop graph where human node embeddings are updated using local joint features via a query-key structure, mimicking human visual attention to specific joints.
• Encodes interaction edges using spatial relationships and human pose features, enabling interactiveness-aware message passing in the graph.
• Employs a query-based attention mechanism that attends to human joint features to refine human node representations, enhancing interaction classification.
• Pre-trains the ViT backbone using CLIP weights to improve feature quality and generalization, especially for few-shot or zero-shot scenarios.

Experimental results

Research questions

• RQ1 Can a Vision Transformer backbone be effectively and efficiently used in two-stage HOI detection without suffering from spatial quantization issues?
• RQ2 How can human pose information be integrated into the interaction prediction process to improve classification accuracy?
• RQ3 Does modeling the interaction recognition process as a three-step human perception pipeline (localization → interactiveness → joint focus) lead to better performance?
• RQ4 Can a self-loop structure in the graph network improve performance by focusing on local human joint features?
• RQ5 Does using CLIP-pretrained weights on ViT improve HOI detection performance compared to ImageNet or COCO pretraining?

Key findings

• ViPLO achieves a new state-of-the-art mAP of 36.97 on the HICO-DET dataset under the Full setting, outperforming previous methods by a significant margin.
• The model achieves a +2.07 mAP improvement over the previous SOTA on HICO-DET, demonstrating the effectiveness of the proposed components.
• The MOA module reduces performance degradation caused by quantization, improving mAP by 1.08 points over a baseline ViT + ROI method.
• The pose-conditioned self-loop graph contributes 0.44 mAP improvement over a baseline without self-loop, validating its role in focusing on relevant joints.
• ViPLO maintains inference speed and GPU memory usage comparable to ResNet-50-based models, with lower memory consumption than SCG using ResNet-50.
• Using CLIP-pretrained ViT weights leads to superior performance compared to COCO-pretrained ResNet, even when using the same pretraining scheme, showing the benefit of CLIP initialization.