[Paper Review] Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition

Jinwoo Choi, Chen Gao+2 more|arXiv (Cornell University)|Dec 11, 2019

Human Pose and Action RecognitionComputer Science99 citations

TL;DR

The paper introduces a debiasing framework for video action recognition that mitigates scene bias by using a scene-adversarial loss and a human-masked entropy loss, improving transfer to classification, localization, and detection tasks.

ABSTRACT

Human activities often occur in specific scene contexts, e.g., playing basketball on a basketball court. Training a model using existing video datasets thus inevitably captures and leverages such bias (instead of using the actual discriminative cues). The learned representation may not generalize well to new action classes or different tasks. In this paper, we propose to mitigate scene bias for video representation learning. Specifically, we augment the standard cross-entropy loss for action classification with 1) an adversarial loss for scene types and 2) a human mask confusion loss for videos where the human actors are masked out. These two losses encourage learning representations that are unable to predict the scene types and the correct actions when there is no evidence. We validate the effectiveness of our method by transferring our pre-trained model to three different tasks, including action classification, temporal localization, and spatio-temporal action detection. Our results show consistent improvement over the baseline model without debiasing.

Motivation & Objective

• Motivate and quantify scene representation bias in action recognition datasets.
• Propose a debiasing training objective to learn scene-invariant features.
• Enable better generalization through transfer learning to multiple action understanding tasks.
• Evaluate the debiasing method across action classification, temporal localization, and spatio-temporal detection.

Proposed method

• Pre-train a CNN on Mini-Kinetics-200 with standard cross-entropy loss for action labels.
• Add a scene adversarial loss to learn scene-invariant features via a scene classifier on top of the feature extractor.
• Add a human mask confusion loss by masking humans in videos and maximizing the entropy of predicted actions for those videos.
• Use a gradient reversal layer to train the scene-adversarial objective in an adversarial fashion.
• Mask humans using an off-the-shelf detector and replace their pixels with frame-mean values during training.
• Fine-tune the debiased representation on downstream tasks for action classification, localization, and detection.

Experimental results

Research questions

• RQ1 Does the proposed debiasing reduce scene representation bias in video datasets?
• RQ2 Do debiased representations transfer better to unseen action classes and tasks beyond pre-training data?
• RQ3 What is the impact of the two proposed debiasing losses on generalization?
• RQ4 How do different pseudo-scene labels affect debiasing effectiveness?

Key findings

• Debiasing reduces scene-dependent features, evidenced by scene classifier accuracy dropping from 29.7% to 2.9% on Mini-Kinetics-200 validation.
• Debiased pre-training consistently improves transfer performance on HMDB-51, UCF-101, and Diving48 for action classification.
• Debiasing also improves temporal action localization on THUMOS-14 and spatio-temporal action detection on JHMDB.
• Using soft pseudo scene labels outperforms hard labels for scene-adversarial training.
• Both LAdv and LEnt contribute to gains; using both yields the best results.