[论文解读] Meta-Learning without Memorization

Mingzhang Yin, George Tucker+3 more|arXiv (Cornell University)|Dec 9, 2019

Domain Adaptation and Few-Shot Learning参考文献 36被引用 72

一句话总结

这篇论文将记忆化识别为元学习中在非互斥任务下的关键失败模式，并提出信息理论元正则化以强制任务数据驱动的适应，在挑战性设置下提高 MAML 和 CNP 的性能。

ABSTRACT

The ability to learn new concepts with small amounts of data is a critical aspect of intelligence that has proven challenging for deep learning methods. Meta-learning has emerged as a promising technique for leveraging data from previous tasks to enable efficient learning of new tasks. However, most meta-learning algorithms implicitly require that the meta-training tasks be mutually-exclusive, such that no single model can solve all of the tasks at once. For example, when creating tasks for few-shot image classification, prior work uses a per-task random assignment of image classes to N-way classification labels. If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes. This requirement means that the user must take great care in designing the tasks, for example by shuffling labels or removing task identifying information from the inputs. In some domains, this makes meta-learning entirely inapplicable. In this paper, we address this challenge by designing a meta-regularization objective using information theory that places precedence on data-driven adaptation. This causes the meta-learner to decide what must be learned from the task training data and what should be inferred from the task testing input. By doing so, our algorithm can successfully use data from non-mutually-exclusive tasks to efficiently adapt to novel tasks. We demonstrate its applicability to both contextual and gradient-based meta-learning algorithms, and apply it in practical settings where applying standard meta-learning has been difficult. Our approach substantially outperforms standard meta-learning algorithms in these settings.

研究动机与目标

• 形式化元学习中的记忆化问题，并将其与标准监督过拟合区分开。
• 提出一个使用信息理论的通用元正则化目标以促进数据驱动的适应。
• 证明元正则化可以由 PAC-Bayes 泛化界给出动机。
• 在非互斥任务中对梯度基和上下文元学习都显示出元正则化方法的显著性能提升。

提出的方法

• 引入一个随机瓶颈 z* 及其目标，使 I(y*; D | z*, θ) 增大以防止记忆化（关于激活的方程）。
• 推导出一个可行的元正则化项，作为 q(z*|x*, θ) 对 r(z*) 的 KL 散度惩罚，从而得到正则化损失（式 3）。
• 提出对元参数 θ 的元正则化，通过对先验的 KL 项来界 I(y1:N, D1:N; θ | x*1:N)（式 4）。
• 将激活和权重两种正则化合并为统一的 MR 目标（式 5），并应用于 MAML（权重）和 CNP（编码器），其算法在附录中描述。
• 通过 PAC-Bayes 边界（定理 1）提供理论基础，给出在某些条件下 MR 提升泛化。
• 展示在非互斥任务下对 MAML（MR-MAML(W)）和 CNP 变体（MR-CNP）的适用性。

实验结果

研究问题

• RQ1 元学习算法与领域中记忆化问题的普遍性有多高？
• RQ2 元正则化能否在非互斥任务分布中缓解记忆化？
• RQ3 所提出的元正则化是否与梯度式和基于上下文的元学习方法兼容？
• RQ4 如同 PAC-Bayes 分析所示，元正则化是否能改善泛化保证？

主要发现

• 记忆化是 MAML 和 CNP 在非互斥任务上的一个重大挑战，有时会导致接近随机的测试表现。
• 元正则化的 MAML 和 CNP（MR-MAML 和 MR-CNP）实现了高效的适应性和强泛化，在非互斥任务上显著优于未正则化的基线。
• 对权重的元正则化（MR-MAML(W)）倾向于在不同学习率设置下稳定收敛到适应解，而基于激活的 MR 对超参数更敏感。
• 通过 PAC-Bayes 分析为 MR 提升泛化界提供理论支持，将权重上的 KL 惩罚与更紧的泛化保证联系起来。