[Paper Review] Activity Driven Weakly Supervised Object Detection

Zhenheng Yang, Dhruv Mahajan+3 more|arXiv (Cornell University)|Apr 2, 2019

Human Pose and Action Recognition52 references4 citations

TL;DR

This paper proposes an activity-driven weakly supervised object detection framework that leverages action labels to improve object localization without bounding box annotations. By modeling spatial priors based on human-object interactions, learning consistent object appearances across actions, and jointly training action and object classifiers, the method achieves a 6% mAP improvement over SOTA on the Charades video dataset.

ABSTRACT

Weakly supervised object detection aims at reducing the amount of supervision required to train detection models. Such models are traditionally learned from images/videos labelled only with the object class and not the object bounding box. In our work, we try to leverage not only the object class labels but also the action labels associated with the data. We show that the action depicted in the image/video can provide strong cues about the location of the associated object. We learn a spatial prior for the object dependent on the action (e.g. "ball" is closer to "leg of the person" in "kicking ball"), and incorporate this prior to simultaneously train a joint object detection and action classification model. We conducted experiments on both video datasets and image datasets to evaluate the performance of our weakly supervised object detection model. Our approach outperformed the current state-of-the-art (SOTA) method by more than 6% in mAP on the Charades video dataset.

Motivation & Objective

• To reduce reliance on expensive bounding box annotations in object detection by leveraging action class labels as weak supervision.
• To improve localization accuracy in weakly supervised detection by exploiting spatial and appearance priors derived from human-object interactions in actions.
• To jointly train object detection and action classification models to enhance feature learning and mutual supervision.
• To evaluate the effectiveness of action-driven supervision on both video and image datasets, including egocentric videos.
• To demonstrate that weak supervision from action labels can significantly reduce the need for strong supervision in object detection.

Proposed method

• The method learns a spatial prior for objects based on human body parts (e.g., hand) involved in actions, modeling object locations as dependent on action context.
• It introduces a joint training framework that simultaneously optimizes for action classification and object detection using shared features and multi-task learning.
• A generic object appearance model is trained across different actions involving the same object, ensuring consistent feature learning.
• The model uses object proposals from RPN and applies a classification head to predict object presence, with supervision derived from action labels.
• It incorporates a weakly supervised loss that encourages correct object proposals to be selected based on action context and spatial priors.
• The approach is extended to egocentric videos by using a center-based spatial prior, as human keypoint detection is not feasible.

Experimental results

Research questions

• RQ1 Can action labels provide meaningful spatial and appearance priors to improve weakly supervised object detection?
• RQ2 How does joint learning of action and object classification enhance detection performance compared to independent training?
• RQ3 To what extent can action-driven supervision reduce the need for bounding box annotations in object detection?
• RQ4 How does the method generalize across different data modalities, including videos and images?
• RQ5 What is the impact of adding limited ground-truth bounding boxes when combined with action-based weak supervision?

Key findings

• The proposed method achieves a 6% mAP improvement over the previous SOTA on the Charades video dataset, demonstrating strong performance in weakly supervised detection.
• On the HICO-DET image dataset, the method outperforms existing weakly supervised approaches, showing generalization beyond video data.
• In the egocentric EPIC-KITCHENS dataset, the method outperforms R*CNN and PCL, benefiting from the implicit spatial prior in first-person videos.
• With only 70% of training samples containing bounding box annotations, the method matches the performance of fully supervised models at 100% supervision on Charades.
• Even with 100% bounding box annotations, the method outperforms fully supervised baselines by 2 mAP points due to the benefits of joint training with action and object classification.
• Ablation studies confirm that each component—spatial prior, joint training, and appearance modeling—contributes significantly to the final performance.