[Paper Review] Expecting the Unexpected: Training Detectors for Unusual Pedestrians with Adversarial Imposters

Shiyu Huang, Deva Ramanan|arXiv (Cornell University)|Mar 18, 2017

Human Pose and Action Recognition2 references4 citations

TL;DR

This paper proposes a GAN-inspired framework that uses a game engine to generate synthetic 'adversarial imposters'—realistic, rare pedestrian scenarios such as children playing or people on skateboards—by training a discriminator to distinguish real from synthetic data. The method significantly improves pedestrian detection performance on rare, dangerous scenarios by fine-tuning detectors on a balanced mix of real data and these high-quality synthetic imposters, achieving state-of-the-art results on the novel Precarious Pedestrian dataset.

ABSTRACT

As autonomous vehicles become an every-day reality, high-accuracy pedestrian detection is of paramount practical importance. Pedestrian detection is a highly researched topic with mature methods, but most datasets focus on common scenes of people engaged in typical walking poses on sidewalks. But performance is most crucial for dangerous scenarios, such as children playing in the street or people using bicycles/skateboards in unexpected ways. Such "in-the-tail" data is notoriously hard to observe, making both training and testing difficult. To analyze this problem, we have collected a novel annotated dataset of dangerous scenarios called the Precarious Pedestrian dataset. Even given a dedicated collection effort, it is relatively small by contemporary standards (around 1000 images). To allow for large-scale data-driven learning, we explore the use of synthetic data generated by a game engine. A significant challenge is selected the right "priors" or parameters for synthesis: we would like realistic data with poses and object configurations that mimic true Precarious Pedestrians. Inspired by Generative Adversarial Networks (GANs), we generate a massive amount of synthetic data and train a discriminative classifier to select a realistic subset, which we deem the Adversarial Imposters. We demonstrate that this simple pipeline allows one to synthesize realistic training data by making use of rendering/animation engines within a GAN framework. Interestingly, we also demonstrate that such data can be used to rank algorithms, suggesting that Adversarial Imposters can also be used for "in-the-tail" validation at test-time, a notoriously difficult challenge for real-world deployment.

Motivation & Objective

• To address the lack of realistic, rare pedestrian scenarios in existing datasets, which hinders training and evaluation of robust pedestrian detectors for autonomous vehicles.
• To develop a scalable method for generating high-fidelity synthetic data that captures the statistical and visual characteristics of rare, dangerous pedestrian behaviors.
• To enable effective transfer learning from synthetic data to real-world deployment by using a discriminative classifier to select only the most realistic synthetic samples—'adversarial imposters'—for training.
• To validate that synthetic adversarial imposters can serve not only as training data but also as a reliable testset for evaluating detector performance on rare, hard-to-observe scenarios.

Proposed method

• A rendering pipeline is used to generate synthetic images from scene parameters (e.g., body poses, object configurations), leveraging a game engine’s physics and lighting fidelity.
• A generative adversarial framework is applied where the generator is fixed as the rendering engine, and the latent space is optimized to produce images that fool a discriminator trained on real Precarious Pedestrian images.
• The discriminator is trained to classify real images from the Precarious Pedestrian dataset versus synthetic images, and the best-performing synthetic images—those that most confuse the discriminator—are selected as 'adversarial imposters'.
• A novel detector, RPN+, is proposed that outputs a dense heatmap of pedestrian detections with regressed bounding boxes per pixel, improving detection on rare, non-standard pedestrian poses.
• The final detector is fine-tuned using a combination of real data and adversarial imposters, with ablation studies showing optimal performance when the imposter set size matches the real training set size.
• The discriminator's training progress is monitored, and multiple versions are used to select imposters at different training stages, improving the quality of the synthetic data over time.

Experimental results

Research questions

• RQ1 Can synthetic data generated via a game engine, when refined through adversarial training, produce realistic and diverse examples of rare pedestrian behaviors such as children playing or people on skateboards?
• RQ2 Does fine-tuning a pedestrian detector on a balanced mix of real data and adversarial imposters lead to improved performance on rare, dangerous pedestrian scenarios compared to training on real data alone?
• RQ3 Can adversarial imposters serve as a reliable synthetic testset for evaluating detector robustness on 'in-the-tail' scenarios where real test data is scarce?
• RQ4 How does the size of the adversarial imposter set and the training stage of the discriminator affect the final detector performance?

Key findings

• The RPN+ detector fine-tuned on real data and adversarial imposters achieved a miss rate of 42.47% at a false positive rate of 10−1, outperforming standard fine-tuning (6% better) and training only on real data (24% better).
• Optimal performance was achieved when the number of adversarial imposters was approximately equal to the size of the real training set (500 images), indicating the importance of balanced data distribution.
• The performance of the final detector improved as the discriminator became more accurate over training, showing that better-quality synthetic data leads to better generalization.
• The ranking of detectors on real test data and on synthetic adversarial imposter test data was identical, suggesting that synthetic data can reliably substitute for real rare-scenario test data.
• The method demonstrated that synthetic data generated through adversarial refinement can effectively close the domain gap between synthetic and real-world pedestrian detection scenarios.
• The RPN+ model trained on adversarial imposters performed poorly on the Caltech dataset, indicating domain shift due to differences in scale, aspect ratio, and scene distribution, highlighting the importance of domain-specific data.