[Paper Review] Dataset Distillation via Factorization

Songhua Liu, Kai Wang+3 more|arXiv (Cornell University)|Oct 30, 2022

Image Processing Techniques and Applications59 citations

TL;DR

HaBa introduces a hallucinator-basis factorization for dataset distillation, enabling expressive synthetic data with fewer parameters and better downstream performance, including cross-architecture gains. It adds adversarial contrastive constraints to boost diversity and informativeness, and is plug-and-play with existing DD baselines.

ABSTRACT

In this paper, we study \xw{dataset distillation (DD)}, from a novel perspective and introduce a \emph{dataset factorization} approach, termed \emph{HaBa}, which is a plug-and-play strategy portable to any existing DD baseline. Unlike conventional DD approaches that aim to produce distilled and representative samples, \emph{HaBa} explores decomposing a dataset into two components: data \emph{Ha}llucination networks and \emph{Ba}ses, where the latter is fed into the former to reconstruct image samples. The flexible combinations between bases and hallucination networks, therefore, equip the distilled data with exponential informativeness gain, which largely increase the representation capability of distilled datasets. To furthermore increase the data efficiency of compression results, we further introduce a pair of adversarial contrastive constraints on the resultant hallucination networks and bases, which increase the diversity of generated images and inject more discriminant information into the factorization. Extensive comparisons and experiments demonstrate that our method can yield significant improvement on downstream classification tasks compared with previous state of the arts, while reducing the total number of compressed parameters by up to 65\%. Moreover, distilled datasets by our approach also achieve extasciitilde10\% higher accuracy than baseline methods in cross-architecture generalization. Our code is available \href{https://github.com/Huage001/DatasetFactorization}{here}.

Motivation & Objective

• Motivate and address data/storage efficiency in dataset distillation (DD).
• Propose a factorization of synthetic data into bases and hallucinators to increase information content.
• Introduce adversarial contrastive constraints to diversify generated data.
• Demonstrate plug-and-play compatibility with existing DD baselines and show improved performance.

Proposed method

• Factorize synthetic data into a set of bases B and hallucinators H such that S = {H_theta_j} ∪ {(x_hat_i, y_hat_i)}.
• Each hallucinator takes a basis as input and outputs a hallucinated image via an encoder-transformer-decoder pipeline with affine scaling and shifting.
• Introduce adversarial contrastive loss L_cos and a (optionally supervised) contrastive loss L_con to maximize diversity and reduce redundancy among samples sharing a basis.
• Incorporate a taskLoss L_task and a DD objective L_DD; train in an alternating, end-to-end differentiable pipeline; HaBa is compatible as a plug-in with existing DD objectives.
• Optionally combine with concurrent efficient data parameterization (IDC) and evaluate cross-architecture generalization.

Experimental results

Research questions

• RQ1 Can HaBa improve downstream model performance compared to state-of-the-art DD baselines under the same storage budget?
• RQ2 Does factorizing into bases and hallucinators increase data diversity and information content without increasing storage?
• RQ3 How does HaBa affect cross-architecture generalization (training on one architecture, evaluating on others)?
• RQ4 What is the impact of adversarial contrastive constraints on performance and diversity?

Key findings

• HaBa yields significant improvements over prior DD methods across SVHN, CIFAR10, and CIFAR100 benchmarks.
• Under the same storage budget, HaBa reduces the total number of compressed parameters by up to 65%.
• HaBa achieves about 10% higher accuracy than baseline methods in cross-architecture generalization scenarios.
• Bases store core structures while hallucinators render diverse styles, increasing data diversity without extra storage.
• HaBa demonstrates consistent gains when built atop multiple DD baselines (DC, DM, MTT) and supports cross-architecture gains with various networks (ConvNet, ResNet, VGG, AlexNet).
• Qualitative visualizations show that different hallucinators produce diverse images from shared bases, enhancing dataset informativeness.