[论文解读] Fraternal Dropout

Konrad Żołna, Devansh Arpit+2 more|arXiv (Cornell University)|Oct 31, 2017

Topic Modeling参考文献 13被引用 8

一句话总结

本文提出了一种新颖的循环神经网络（RNN）正则化技术—— fraternal dropout，该方法通过在训练过程中使用不同的dropout掩码来训练两个参数共享的相同RNN，同时最小化它们在softmax前预测结果的差异，从而提升对dropout变化的鲁棒性，并在序列建模任务（包括Penn Treebank和Wikitext-2）上取得了最先进性能，尤其在图像字幕生成和半监督学习任务中表现显著。

ABSTRACT

Recurrent neural networks (RNNs) are important class of architectures among neural networks useful for language modeling and sequential prediction. However, optimizing RNNs is known to be harder compared to feed-forward neural networks. A number of techniques have been proposed in literature to address this problem. In this paper we propose a simple technique called fraternal dropout that takes advantage of dropout to achieve this goal. Specifically, we propose to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions. In this way our regularization encourages the representations of RNNs to be invariant to dropout mask, thus being robust. We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout. We evaluate our model and achieve state-of-the-art results in sequence modeling tasks on two benchmark datasets - Penn Treebank and Wikitext-2. We also show that our approach leads to performance improvement by a significant margin in image captioning (Microsoft COCO) and semi-supervised (CIFAR-10) tasks.

研究动机与目标

• 解决与前馈网络相比，优化循环神经网络（RNNs）的困难。
• 减少基于dropout的正则化中训练阶段与推理阶段之间的差异。
• 通过鼓励在不同dropout掩码下表示的不变性，提升RNN的泛化能力和鲁棒性。
• 在标准序列建模基准和迁移任务上实现最先进性能。

提出的方法

• 在训练过程中，使用共享参数但不同dropout掩码的两个相同RNN进行训练。
• 通过最小化两个RNN在softmax前预测结果的L2差异，以增强鲁棒性。
• 将正则化形式化为一个上界约束，该约束控制期望线性dropout目标。
• 利用所得损失来稳定训练过程，并减小训练与推理之间的差距。
• 将该方法应用于语言建模、图像字幕生成和半监督学习任务。

实验结果

研究问题

• RQ1 使用不同dropout掩码训练两个相同的RNN是否能提升泛化能力和鲁棒性？
• RQ2 该方法是否能减少基于dropout模型在训练与推理阶段之间的差异？
• RQ3 fraternal dropout是否能在序列建模基准上实现最先进性能？
• RQ4 该方法是否能推广到非语言任务，如图像字幕生成和半监督学习？

主要发现

• fraternal dropout在Penn Treebank和Wikitext-2语言建模基准上取得了最先进结果。
• 该方法在Microsoft COCO图像字幕生成任务上显著提升了性能。
• 在CIFAR-10的半监督学习任务中，该方法带来了显著性能增益。
• 正则化项被期望线性dropout目标所上界控制，确保了理论上的稳定性。