[Paper Review] A Comprehensive Survey of Dataset Distillation

Shiye Lei, Dacheng Tao|arXiv (Cornell University)|Jan 13, 2023

Machine Learning and Data Classification106 references4 citations

TL;DR

This survey provides a comprehensive overview of dataset distillation (DD), a technique that synthesizes small, high-informative datasets from large real datasets to accelerate deep learning training. It categorizes DD methods into meta-learning and data matching frameworks, analyzes factorized approaches using latent codes, and evaluates performance, applications, and challenges such as cross-architecture transferability, privacy, and robustness.

ABSTRACT

Deep learning technology has developed unprecedentedly in the last decade and has become the primary choice in many application domains. This progress is mainly attributed to a systematic collaboration in which rapidly growing computing resources encourage advanced algorithms to deal with massive data. However, it has gradually become challenging to handle the unlimited growth of data with limited computing power. To this end, diverse approaches are proposed to improve data processing efficiency. Dataset distillation, a dataset reduction method, addresses this problem by synthesizing a small typical dataset from substantial data and has attracted much attention from the deep learning community. Existing dataset distillation methods can be taxonomized into meta-learning and data matching frameworks according to whether they explicitly mimic the performance of target data. Although dataset distillation has shown surprising performance in compressing datasets, there are still several limitations such as distilling high-resolution data or data with complex label spaces. This paper provides a holistic understanding of dataset distillation from multiple aspects, including distillation frameworks and algorithms, factorized dataset distillation, performance comparison, and applications. Finally, we discuss challenges and promising directions to further promote future studies on dataset distillation.

Motivation & Objective

• To provide a systematic and holistic understanding of dataset distillation (DD) across frameworks, algorithms, and applications.
• To analyze the limitations of existing DD methods, including poor cross-architecture generalization, privacy risks, and vulnerability to adversarial attacks.
• To explore factorized DD approaches that use latent codes and decoders to improve compression ratio and performance.
• To evaluate the trade-offs between efficiency, generalization, and robustness in distilled datasets.
• To identify open challenges and future research directions in efficient, private, and robust dataset distillation.

Proposed method

• Categorizes DD methods into two main frameworks: meta-learning (optimizing synthetic data via nested optimization of model risk) and data matching (matching gradients, trajectories, or distributions of real data).
• Reviews factorized DD techniques that decompose synthetic data generation into latent codes and decoders, enabling higher compression and better performance.
• Analyzes optimization strategies such as backpropagation through time, kernel ridge regression, and GAN-based generators for synthetic data creation.
• Evaluates performance using metrics like test accuracy, generalization gap, and transferability across different architectures.
• Examines privacy-preserving mechanisms, including differential privacy and gradient noise injection, to protect sensitive data in distilled sets.
• Investigates robustness enhancement via distillation of adversarial examples and pseudo-outliers for OoD detection.

Experimental results

Research questions

• RQ1 How do meta-learning and data matching frameworks differ in their optimization objectives and performance in dataset distillation?
• RQ2 To what extent can factorized DD using latent codes improve compression ratio and generalization compared to direct pixel-space optimization?
• RQ3 Why does distilled data exhibit poor transferability across different neural network architectures, and how can this be mitigated?
• RQ4 What are the privacy risks in dataset distillation, and how can differential privacy be effectively integrated into the distillation pipeline?
• RQ5 How can dataset distillation be leveraged to improve model robustness against adversarial attacks and out-of-distribution inputs?

Key findings

• Factorized DD using latent codes and decoders significantly improves generalization and compression ratio by capturing intraclass variations more effectively than pixel-space optimization.
• Direct optimization in pixel space leads to poor cross-architecture transferability due to overfitting to the specific architecture used in distillation.
• Regularizing distillation through latent code optimization (e.g., using pretrained GANs) enhances cross-architecture performance by generating more realistic synthetic data.
• Private dataset distillation can achieve better privacy-accuracy trade-offs when combined with DP-SGD and gradient matching, though theoretical claims require careful validation.
• DD can effectively distill adversarial examples and pseudo-outliers, enabling downstream models to achieve improved robustness and OoD detection without full adversarial training.
• Models trained on distilled data often suffer from overconfidence and poor calibration, which can be mitigated by masking synthetic data during distillation to preserve semantic completeness.