[Paper Review] Do We Need More Training Data?

Xiangxin Zhu, Carl Vondrick+2 more|Mar 5, 2015

Electronic Health Records Systems3 citations

TL;DR

This paper investigates whether increasing training data alone improves object detection performance in HOG-based template models, finding that standard mixture models saturate quickly due to limited model capacity and data noise. It proposes an exemplar deformable part model (EDPM) that uses compositional, parameter-shared parts to extrapolate to unseen templates, achieving significantly better performance than non-parametric mixtures even with limited data, demonstrating that model architecture and parameter sharing are more critical than data volume alone.

ABSTRACT

Datasets for training object recognition systems are steadily increasing in size. This paper investigates the question of whether existing detectors will continue to improve as data grows, or saturate in performance due to limited model complexity and the Bayes risk associated with the feature spaces in which they operate. We focus on the popular paradigm of discriminatively trained templates defined on oriented gradient features. We investigate the performance of mixtures of templates as the number of mixture components and the amount of training data grows. Surprisingly, even with proper treatment of regularization and "outliers", the performance of classic mixture models appears to saturate quickly ($\sim$10 templates and $\sim$100 positive training examples per template). This is not a limitation of the feature space as compositional mixtures that share template parameters via parts and that can synthesize new templates not encountered during training yield significantly better performance. Based on our analysis, we conjecture that the greatest gains in detection performance will continue to derive from improved representations and learning algorithms that can make efficient use of large datasets.

Motivation & Objective

• To determine whether increasing training data continues to improve performance in HOG-based object detectors beyond current dataset scales.
• To diagnose why standard mixture models of templates saturate quickly despite growing training data.
• To evaluate whether current methods for sub-category discovery (e.g., clustering aspect ratios or appearance) are sufficient for leveraging larger datasets.
• To propose a non-parametric extension of deformable part models (EDPM) that enables better generalization and performance scaling with data.
• To demonstrate that improved model inductive biases—especially compositional, parameter-shared representations—are more impactful than simply increasing data volume.

Proposed method

• The authors train linear SVMs on HOG features using a large-scale dataset, an order of magnitude larger than PASCAL VOC, to study performance as a function of data size and model complexity.
• They evaluate standard non-parametric mixture models of templates, varying the number of components and training examples, and observe early performance saturation.
• They introduce the exemplar deformable part model (EDPM), a non-parametric extension of DPMs that shares part parameters across templates and enables composition of new configurations from training exemplars.
• EDPM uses a part-based, compositional structure to implicitly represent an exponentially large set of templates, allowing efficient nearest-neighbor-style inference without explicit enumeration.
• They apply cross-validation to regularize model complexity based on dataset size, correcting for overfitting and performance degradation with more data.
• They compare EDPM to standard DPMs and non-parametric mixtures, showing that parameter sharing and compositionality enable better generalization and scalability.

Experimental results

Research questions

• RQ1 Does performance of HOG-based template detectors continue to improve with increasing training data, or does it saturate due to model limitations?
• RQ2 Why do standard non-parametric mixture models of templates fail to scale effectively with more data, even with regularization?
• RQ3 Can compositional, parameter-shared models like DPMs outperform non-parametric mixtures when generalizing to unseen object configurations?
• RQ4 To what extent do manual or heuristic sub-category labeling strategies (e.g., clustering aspect ratios) limit performance gains from larger datasets?
• RQ5 Can non-parametric, compositional models like EDPM make better use of large datasets than traditional mixture models by enabling extrapolation to unseen templates?

Key findings

• Standard non-parametric mixture models of HOG templates saturate in performance after only ~10 components and ~100 positive examples per component, despite growing training data.
• Performance degradation with more data is observed in off-the-shelf implementations due to inadequate regularization, which can be corrected via cross-validation.
• Deformable part models (DPMs) achieve better performance than non-parametric mixtures by sharing part parameters across templates, effectively encoding an exponentially large set of configurations.
• The exemplar deformable part model (EDPM) significantly outperforms both non-parametric mixtures and standard DPMs by enabling composition of new templates from shared parts, even with limited training data.
• EDPM achieves high performance with only 100–200 positive examples per category, suggesting that HOG templates have limited effective capacity and do not require thousands of examples.
• The study concludes that future gains in detection performance will stem more from improved model inductive biases—especially compositionality and parameter sharing—than from simply increasing training data size.