[Paper Review] Dataset Condensation with Gradient Matching

Bo Zhao, Konda Reddy Mopuri+1 more|arXiv (Cornell University)|Jun 10, 2020

Machine Learning and Data Classification63 references105 citations

TL;DR

The paper learns a small synthetic dataset by matching gradients to a large real dataset, enabling training from scratch with far fewer samples while preserving performance across architectures.

ABSTRACT

As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes a training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained on the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods. Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.

Motivation & Objective

• Motivate data-efficient learning by condensing large datasets into small, informative synthetic sets.
• Formulate dataset condensation as gradient matching between real-data and synthetic-data training losses.
• Develop a curriculum gradient matching approach to enable scalable optimization on modern deep nets.
• Demonstrate that condensed data can train from scratch with competitive generalization across architectures.

Proposed method

• Define a condensed set S of synthetic samples with labels to train neural networks.
• Formulate a gradient-matching objective D(·,·) between gradients of L^S and L^T w.r.t. network parameters.
• Use a curriculum gradient matching strategy that tracks optimizer steps to avoid expensive unrolling.
• Apply back-optimization with a fixed-step inner loop to update S via minimizing gradient-distance across layers.
• Train networks on S to approximate training on the full dataset, enabling cross-architecture generalization.
• Decompose the gradient-matching distance into layer-wise terms with a per-output-node grouping for FC and Conv layers (Eq. 10).

Experimental results

Research questions

• RQ1 Can a large image dataset be compressed into a small synthetic set without substantial loss in downstream performance?
• RQ2 Can a network trained on the synthetic set generalize to real images and across different architectures?
• RQ3 Can a single condensed set be effective for training multiple network architectures?
• RQ4 Is gradient matching a scalable, memory-efficient alternative to inner-loop unrolling for dataset condensation?

Key findings

• The condensed set outperforms coreset baselines (Random, Herding, K-Center, Forgetting) and DD on MNIST, FashionMNIST, SVHN, and CIFAR-10 across several architectures.
• With 50 images per class on MNIST, the method reaches 99.6% of the full-dataset upper bound (99.6% vs 99.6%? note: source states 99.6% for whole dataset upper bound and 98.8% with 50 images per class).