[Paper Review] Dataset Distillation: A Comprehensive Review

Ruonan Yu, Songhua Liu+1 more|arXiv (Cornell University)|Jan 17, 2023

Machine Learning and Data Classification4 citations

TL;DR

This paper presents a comprehensive review of dataset distillation (DD), a technique that generates small, synthetic datasets to train deep learning models with performance comparable to those trained on large original datasets. It categorizes existing DD methods into three optimization-based streams—performance matching, parameter matching, and distribution matching—and provides a unified algorithmic framework, while identifying key challenges in generalization, scalability, and privacy.

ABSTRACT

Recent success of deep learning is largely attributed to the sheer amount of data used for training deep neural networks.Despite the unprecedented success, the massive data, unfortunately, significantly increases the burden on storage and transmission and further gives rise to a cumbersome model training process. Besides, relying on the raw data for training \emph{per se} yields concerns about privacy and copyright. To alleviate these shortcomings, dataset distillation~(DD), also known as dataset condensation (DC), was introduced and has recently attracted much research attention in the community. Given an original dataset, DD aims to derive a much smaller dataset containing synthetic samples, based on which the trained models yield performance comparable with those trained on the original dataset. In this paper, we give a comprehensive review and summary of recent advances in DD and its application. We first introduce the task formally and propose an overall algorithmic framework followed by all existing DD methods. Next, we provide a systematic taxonomy of current methodologies in this area, and discuss their theoretical interconnections. We also present current challenges in DD through extensive experiments and envision possible directions for future works.

Motivation & Objective

• To systematically survey and categorize recent advances in dataset distillation (DD) and its applications.
• To identify and analyze the core methodological differences among existing DD approaches through a unified algorithmic framework.
• To evaluate the performance, generalization, and scalability of DD methods across diverse datasets and architectures.
• To highlight open challenges in transferability, privacy, and applicability beyond image classification.
• To envision future research directions in efficient learning, continual learning, and secure data synthesis.

Proposed method

• Proposes a unified algorithmic framework for all existing DD methods, abstracting key components such as synthetic data initialization, optimization objectives, and training procedures.
• Classifies DD methods into three main categories based on optimization objectives: performance matching (e.g., DSA), parameter matching (e.g., DM), and distribution matching (e.g., FRePo).
• Introduces theoretical analysis to reveal interconnections among the three optimization strategies, showing that they are related through shared inductive biases and optimization dynamics.
• Employs a two-stage training paradigm in some methods: first, synthetic data is optimized to mimic real data behavior; second, the model is trained on the synthetic data to evaluate performance.
• Uses gradient-based optimization to iteratively update synthetic samples, minimizing the loss between model predictions on synthetic data and those on real data.
• Applies architectural regularization and normalization techniques (e.g., batch norm unbinding) to improve transferability of synthetic data across different network architectures.

Experimental results

Research questions

• RQ1 How do different optimization objectives in dataset distillation—performance, parameter, and distribution matching—compare in terms of model performance and generalization?
• RQ2 What are the theoretical relationships and trade-offs among the three main streams of DD methods?
• RQ3 To what extent can synthetic datasets generalize across different deep neural network architectures and tasks?
• RQ4 How do hyperparameters such as the number of synthetic samples (IPC) and training epochs affect the performance and scalability of DD?
• RQ5 What are the key challenges in applying DD to real-world scenarios involving privacy, security, and non-classification tasks?

Key findings

• On CIFAR-10 with 10 images per class, FRePo achieves 47.4% top-1 accuracy on ResNet18, outperforming earlier methods like DD (33.9%) and DC (43.3%).
• On ImageNet-1K with 100 images per class, TESLA achieves 27.9% top-1 accuracy, approaching the performance of models trained on the full dataset.
• Synthetic datasets optimized for a specific architecture show poor transferability to other architectures, especially when training and synthetic data are tightly coupled.
• Methods like DM and IDC improve transferability by decoupling the network from the synthetic data optimization process, though at the cost of reduced performance or increased computation.
• The choice of normalization layers and training strategies significantly impacts the generalization of synthetic data across architectures.
• Current DD methods struggle with generalization to complex tasks such as semantic segmentation and object detection, indicating a need for task-specific design in future work.