[Paper Review] Hide-and-Seek: A Data Augmentation Technique for Weakly-Supervised Localization and Beyond

Krishna Kumar Singh, Hao Yu+3 more|arXiv (Cornell University)|Nov 6, 2018

Human Pose and Action Recognition37 references68 citations

TL;DR

Hide-and-Seek hides random image patches during training to force networks to learn multiple object parts, improving weakly-supervised localization and generalizing to various vision tasks without changing architectures.

ABSTRACT

We propose 'Hide-and-Seek' a general purpose data augmentation technique, which is complementary to existing data augmentation techniques and is beneficial for various visual recognition tasks. The key idea is to hide patches in a training image randomly, in order to force the network to seek other relevant content when the most discriminative content is hidden. Our approach only needs to modify the input image and can work with any network to improve its performance. During testing, it does not need to hide any patches. The main advantage of Hide-and-Seek over existing data augmentation techniques is its ability to improve object localization accuracy in the weakly-supervised setting, and we therefore use this task to motivate the approach. However, Hide-and-Seek is not tied only to the image localization task, and can generalize to other forms of visual input like videos, as well as other recognition tasks like image classification, temporal action localization, semantic segmentation, emotion recognition, age/gender estimation, and person re-identification. We perform extensive experiments to showcase the advantage of Hide-and-Seek on these various visual recognition problems.

Motivation & Objective

• Introduce a general-purpose data augmentation technique that complements existing methods.
• Improve object localization in weakly-supervised settings without additional annotations.
• Demonstrate applicability of the method across multiple tasks and architectures.

Proposed method

• Divide each training image into an S×S patch grid and hide each patch with probability p_hide during training.
• Set hidden pixel values to the dataset mean to align training and testing activation distributions.
• Apply the technique across CNNs (e.g., AlexNet, GoogLeNet) and use CAM/GAP for localization.
• Extend the approach to videos by hiding frame segments during training for temporal action localization.
• Evaluate on tasks including weakly-supervised object localization, semantic segmentation, temporal action localization, and more.

Experimental results

Research questions

• RQ1 Can random patch hiding during training improve localization beyond standard augmentation?
• RQ2 Is Hide-and-Seek effective across multiple architectures and vision tasks?
• RQ3 How should hidden patch values be set to minimize train–test distribution mismatch?
• RQ4 Can the method be extended from images to videos for temporal localization?
• RQ5 What is the impact of patch size and variability on performance?

Key findings

• Significant localization gains over baselines on ILSVRC 2016 for GT-known Loc and Top-1 Loc with multiple patch sizes.
• GoogLeNet-HaS outperforms GoogLeNet-GAP across all tested patch sizes for localization metrics.
• AlexNet-HaS and GoogLeNet-HaS achieve improvements of several points in localization metrics compared to the full-image baselines.
• The mixed-size HaS variant (HaS-Mixed) yields the best Top-1 Loc on AlexNet.
• Hide-and-Seek improves performance across a range of tasks beyond localization, including image classification, semantic segmentation, emotion recognition, and re-identification.