[论文解读] Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching

Ziyao Guo, Kai Wang+4 more|arXiv (Cornell University)|Oct 9, 2023

Time Series Analysis and Forecasting被引用 4

一句话总结

本文提出了一种新颖的数据集提炼方法——难度对齐轨迹匹配（Difficulty-Aligned Trajectory Matching, DATM），该方法动态地将合成数据模式的难度与合成数据集的规模对齐。通过在不同阶段匹配训练轨迹——低IPC时匹配早期阶段，高IPC时匹配晚期阶段，DATM在完整数据集规模下于CIFAR-10上实现了无损性能，克服了以往轨迹匹配方法在IPC增加时出现的性能崩溃问题。

ABSTRACT

The ultimate goal of Dataset Distillation is to synthesize a small synthetic dataset such that a model trained on this synthetic set will perform equally well as a model trained on the full, real dataset. Until now, no method of Dataset Distillation has reached this completely lossless goal, in part due to the fact that previous methods only remain effective when the total number of synthetic samples is extremely small. Since only so much information can be contained in such a small number of samples, it seems that to achieve truly loss dataset distillation, we must develop a distillation method that remains effective as the size of the synthetic dataset grows. In this work, we present such an algorithm and elucidate why existing methods fail to generate larger, high-quality synthetic sets. Current state-of-the-art methods rely on trajectory-matching, or optimizing the synthetic data to induce similar long-term training dynamics as the real data. We empirically find that the training stage of the trajectories we choose to match (i.e., early or late) greatly affects the effectiveness of the distilled dataset. Specifically, early trajectories (where the teacher network learns easy patterns) work well for a low-cardinality synthetic set since there are fewer examples wherein to distribute the necessary information. Conversely, late trajectories (where the teacher network learns hard patterns) provide better signals for larger synthetic sets since there are now enough samples to represent the necessary complex patterns. Based on our findings, we propose to align the difficulty of the generated patterns with the size of the synthetic dataset. In doing so, we successfully scale trajectory matching-based methods to larger synthetic datasets, achieving lossless dataset distillation for the very first time. Code and distilled datasets are available at https://gzyaftermath.github.io/DATM.

研究动机与目标

• 解决现有数据集提炼方法在合成数据集规模增大时性能下降的局限性。
• 探究基于轨迹匹配的方法在高图像每类数（IPC）设置下失效的原因。
• 识别出在提炼过程中学习到的模式难度（简单 vs. 复杂）必须与合成数据集的规模相匹配。
• 开发一种方法，在低IPC和高IPC两种场景下均保持高性能，实现无损提炼。
• 证明轨迹匹配的训练阶段选择（早期 vs. 晚期）对提炼质量具有关键影响，具体取决于IPC水平。

提出的方法

• 提出难度对齐轨迹匹配（DATM），根据目标IPC选择教师网络的特定训练阶段（早期或晚期）进行匹配。
• 利用轨迹匹配优化合成数据，使学生模型在合成数据上的训练动态与在真实数据上的训练动态相匹配。
• 通过选择匹配早期阶段（简单模式）或晚期阶段（复杂模式）的轨迹来控制生成模式的难度。
• 根据目标IPC动态切换早期或晚期轨迹匹配：低IPC时使用早期匹配，高IPC时使用晚期匹配。
• 利用洞察：在高IPC设置下，复杂模式对于覆盖长尾和复杂数据分布至关重要。
• 验证随着IPC增加，使用更深、容量更高的网络（如ResNet18）作为提炼网络变得更加有效，这与低IPC场景下的表现相反。

Figure 1: (a) Illustration of the objective of dataset distillation. (b) The optimization in dataset distillation can be viewed as the process of generating informative patterns on the synthetic dataset. (c) We align the difficulty of the synthetic patterns with the size of the distilled dataset, to

实验结果

研究问题

• RQ1 为何现有基于轨迹匹配的数据集提炼方法在合成数据集规模增大时无法有效扩展？
• RQ2 在提炼过程中学习到的模式难度（简单 vs. 复杂）与合成数据集规模之间有何关系？
• RQ3 能否通过将合成模式的难度与合成数据集规模对齐，来提升所有IPC场景下的提炼性能？
• RQ4 在不同IPC水平下，使用不同架构的提炼网络（如ConvNet与ResNet18）对提炼质量有何影响？
• RQ5 是否可能在所有IPC值下实现真正的无损提炼——即在合成数据上训练的模型性能与在完整真实数据集上训练的模型性能完全一致？

主要发现

• 以往的轨迹匹配方法在高IPC下失效，是因为默认学习的是简单模式，导致合成数据向均值坍塌，丢失了长尾分布细节。
• 匹配晚期轨迹（即学习复杂模式的阶段）在高IPC设置下显著提升了性能，实现了对复杂和稀有数据模式的有效覆盖。
• DATM在CIFAR-10上即使在原始数据集100%的规模下也实现了无损性能，其测试准确率与在完整真实数据集上训练的模型完全一致。
• 在低IPC（如1%）时，DATM优于先前的最先进方法，保持了与MTT及其他领先方法相当的高准确率。
• 在高IPC下，ResNet18作为提炼网络的性能与ConvNet相当或更优，证实了在数据充足时，更高容量的网络更具有效性。
• 先前方法的性能随IPC增加而低于随机选择水平，而DATM始终保持有效，甚至在某些情况下超越完整数据集，表明其具备鲁棒性与可扩展性。

Figure 2: We train expert models on CIFAR-10 for 40 epochs. Then the distillation is performed under different IPC settings by matching either early trajectories $\{\theta^{*}_{t}|0\leq t\leq 20\}$ , late trajectories $\{\theta^{*}_{t}|20\leq t\leq 40\}$ , or all trajectories $\{\theta^{*}_{t}|0\leq