[Paper Review] Is Object Detection Necessary for Human-Object Interaction Recognition?

Ying Jin, Yinpeng Chen+5 more|arXiv (Cornell University)|Jul 27, 2021

Multimodal Machine Learning Applications36 references4 citations

TL;DR

This paper proposes DEFR, a detection-free human-object interaction (HOI) recognition method that achieves state-of-the-art performance by leveraging CLIP-based text embedding initialization and a novel Log-Sum-Exp Sign (LSE-Sign) loss function. Without relying on object or human keypoint detection, DEFR attains 60.5 mAP on HICO, surpassing detection-supervised methods by 13.4 mAP, demonstrating that image-text pre-training and structured multi-label learning can eliminate detection supervision while improving accuracy.

ABSTRACT

This paper revisits human-object interaction (HOI) recognition at image level without using supervisions of object location and human pose. We name it detection-free HOI recognition, in contrast to the existing detection-supervised approaches which rely on object and keypoint detections to achieve state of the art. With our method, not only the detection supervision is evitable, but superior performance can be achieved by properly using image-text pre-training (such as CLIP) and the proposed Log-Sum-Exp Sign (LSE-Sign) loss function. Specifically, using text embeddings of class labels to initialize the linear classifier is essential for leveraging the CLIP pre-trained image encoder. In addition, LSE-Sign loss facilitates learning from multiple labels on an imbalanced dataset by normalizing gradients over all classes in a softmax format. Surprisingly, our detection-free solution achieves 60.5 mAP on the HICO dataset, outperforming the detection-supervised state of the art by 13.4 mAP

Motivation & Objective

• To investigate whether object detection is necessary for high-performance human-object interaction recognition.
• To simplify the HOI recognition pipeline by eliminating detection supervision for both objects and human keypoints.
• To improve detection-free HOI recognition through effective use of image-text pre-training and a novel multi-label loss function.
• To demonstrate that detection-free methods can outperform detection-supervised state-of-the-art approaches.

Proposed method

• Uses a vision transformer backbone pre-trained on ImageNet or CLIP, with the linear classifier head initialized using text embeddings of HOI class labels.
• Employs CLIP's text encoder to generate class label embeddings for classifier initialization, which significantly boosts performance even on image-only pre-trained backbones.
• Introduces the Log-Sum-Exp Sign (LSE-Sign) loss function to normalize gradients across all classes in a softmax-like manner, enabling better learning from multiple labels on imbalanced datasets.
• Applies the LSE-Sign loss during fine-tuning to encourage attention on the most problematic (highest-loss) classes, improving generalization across all HOI categories.
• Utilizes a scalar hyperparameter γ in the LSE-Sign loss to control gradient magnitude, with optimal performance at γ=100.
• Employs a two-stage training pipeline: first, initialize the classifier with pre-encoded text embeddings; second, fine-tune the entire model using the LSE-Sign loss.

Experimental results

Research questions

• RQ1 Can human-object interaction recognition be effectively performed without object or human keypoint detection supervision?
• RQ2 How does image-text pre-training, such as CLIP, improve detection-free HOI recognition when used for classifier initialization?
• RQ3 Can a novel loss function outperform standard binary cross entropy in multi-label HOI recognition with class imbalance?
• RQ4 Does the combination of text embedding initialization and a structured multi-label loss lead to superior performance compared to detection-supervised methods?

Key findings

• DEFR achieves 60.5 mAP on the HICO dataset using only image-text pre-training and the LSE-Sign loss, outperforming the detection-supervised state of the art (47.1 mAP) by 13.4 mAP.
• Classifier initialization using CLIP’s text encoder boosts performance by 23.7 mAP over random initialization when using a CLIP-pretrained ViT-B/32 backbone.
• The LSE-Sign loss outperforms binary cross entropy, weighted BCE, and focal loss, achieving 60.5 mAP compared to 57.9 mAP with BCE on the best model.
• The optimal scalar γ in the LSE-Sign loss is 100, yielding 60.5 mAP; performance drops significantly at higher or lower values.
• Embedding initialization with BERT or CLIP text encoders improves performance by 10+ mAP even when using ImageNet-pretrained image backbones.
• The method achieves state-of-the-art results across all tested pre-training backbones, including CLIP, ImageNet-1K, and ImageNet-21K.