[论文解读] Joint Training of Deep Auto-Encoders

Yingbo Zhou, Devansh Arpit+2 more|arXiv (Cornell University)|May 6, 2014

Generative Adversarial Networks and Image Synthesis参考文献 15被引用 2

一句话总结

本文提出通过在所有层上联合优化单一全局重构目标，对深度自编码器进行联合训练，将每一层自编码器视为局部正则化器。该方法在正则化条件下显著提升了数据建模能力和高层表示学习，尤其在更深的网络架构中，其性能优于贪婪预训练方法，在无监督和有监督设置下均表现更优。

ABSTRACT

Traditionally, when generative models of data are developed via deep architectures, greedy layer-wise pre-training is employed. In a well-trained model, the lower layer of the architecture models the data distribution conditional upon the hidden variables, while the higher layers model the hidden distribution prior. But due to the greedy scheme of the layerwise training technique, the parameters of lower layers are fixed when training higher layers. This makes it extremely challenging for the model to learn the hidden distribution prior, which in turn leads to a suboptimal model for the data distribution. We therefore investigate joint training of deep autoencoders, where the architecture is viewed as one stack of two or more single-layer autoencoders. A single global reconstruction objective is jointly optimized, such that the objective for the single autoencoders at each layer acts as a local, layer-level regularizer. We empirically evaluate the performance of this joint training scheme and observe that it not only learns a better data model, but also learns better higher layer representations, which highlights its potential for unsupervised feature learning. In addition, we find that the usage of regularizations in the joint training scheme is crucial in achieving good performance. In the supervised setting, joint training also shows superior performance when training deeper models. The joint training framework can thus provide a platform for investigating more efficient usage of different types of regularizers, especially in light of the growing volumes of available unlabeled data.

研究动机与目标

• 为解决深度自编码器中贪婪分层预训练的局限性，即在训练高层时固定低层参数的问题。
• 改进隐藏分布先验的学习，因为在贪婪训练中由于低层固定，其学习效果不佳。
• 探究联合优化所有层是否能带来更优的数据分布建模与表征学习效果。
• 评估正则化在联合训练框架中的作用，以提升性能。
• 将联合训练的优势扩展至更深模型的有监督学习任务中。

提出的方法

• 深度自编码器被构建为单层自编码器的堆叠结构，各层之间共享权重。
• 通过联合优化所有层的单一全局重构目标，取代传统的贪婪分层训练过程。
• 每一层的自编码器目标作为整体训练目标的局部正则化器。
• 在联合训练框架中显式应用正则化技术，以稳定学习过程并提升性能。
• 采用端到端反向传播进行训练，使梯度能同时通过所有层。
• 在更深的网络架构上，于无监督和有监督设置下对框架进行了评估。

实验结果

研究问题

• RQ1 与贪婪预训练相比，深度自编码器的联合训练是否能实现更优的数据分布建模？
• RQ2 联合训练能否提升高层表示的质量，从而增强无监督特征学习的效果？
• RQ3 在联合训练框架中，正则化对实现良好性能有多关键？
• RQ4 联合训练是否能泛化到有监督学习任务中，尤其是在更深的模型上？
• RQ5 联合训练框架能否作为探索高效正则化策略的平台？

主要发现

• 与贪婪预训练相比，联合训练显著提升了数据建模性能，通过更有效地学习隐藏分布先验实现。
• 该方法学习到更有效的高层表示，展现出强大的无监督特征学习潜力。
• 正则化在联合训练方案中至关重要，若缺失将导致性能下降。
• 在有监督设置下，联合训练在更深模型上取得了更优结果，超越了贪婪预训练。
• 联合训练框架能够更高效地利用各类正则化方法，尤其在大规模无标签数据下表现更优。
• 结合层级别正则化的全局重构目标，使所有层的表征更加一致且准确。