[Paper Review] Marginalized Denoising Autoencoders for Domain Adaptation

Minmin Chen, Zhixiang Xu+2 more|arXiv (Cornell University)|Jun 18, 2012

Topic Modeling26 references520 citations

TL;DR

This paper proposes marginalized denoising autoencoders (mSDA), a scalable alternative to stacked denoising autoencoders (SDA) for domain adaptation. By marginalizing noise during training, mSDA computes parameters in closed-form, eliminating the need for stochastic optimization and accelerating training by two orders of magnitude while maintaining SDA-level performance on benchmark sentiment analysis tasks.

ABSTRACT

Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters ? in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB^{TM}, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks.

Motivation & Objective

• To address the high computational cost and poor scalability of stacked denoising autoencoders (SDAs) in high-dimensional settings.
• To enable efficient, scalable representation learning for domain adaptation without relying on iterative optimization.
• To develop a method that maintains the robustness of denoising autoencoders while drastically reducing training time.
• To achieve performance comparable to SDA on standard benchmark tasks, particularly in cross-domain sentiment analysis.

Proposed method

• mSDA introduces a marginalized denoising autoencoder that integrates out the noise during training, avoiding the need for sampling during optimization.
• The method computes model parameters in closed-form by analytically marginalizing over the noise distribution, enabling direct computation without iterative optimization.
• The architecture uses a stack of autoencoders with tied weights, where the encoder and decoder are trained jointly using the marginalized reconstruction objective.
• Noise is applied to the input layer, and the model learns to reconstruct the original clean input by minimizing the expected reconstruction error over the noise distribution.
• The closed-form solution allows implementation in just 20 lines of MATLAB, significantly improving training speed.
• The approach preserves the robustness of denoising autoencoders by learning invariant representations through corruption and reconstruction.

Experimental results

Research questions

• RQ1 Can we eliminate the need for iterative optimization in denoising autoencoders while preserving their robustness?
• RQ2 Does marginalizing over noise lead to comparable performance to standard SDA in domain adaptation tasks?
• RQ3 Can the proposed method scale efficiently to high-dimensional features, such as those in text classification?
• RQ4 How much faster is the closed-form parameter computation compared to stochastic gradient descent in SDA?
• RQ5 Is the representation quality of mSDA sufficient for state-of-the-art performance on benchmark domain adaptation tasks?

Key findings

• mSDA achieves training speedups of up to two orders of magnitude compared to standard SDA, reducing training time significantly.
• The method attains nearly identical accuracy to SDA on standard benchmark tasks, including cross-domain sentiment analysis.
• The closed-form parameter computation enables a compact implementation of only 20 lines of MATLAB code.
• The marginalized denoising approach maintains the robustness of SDA by learning invariant representations through noise injection and reconstruction.
• mSDA effectively handles high-dimensional features, demonstrating scalability where traditional SDA struggles due to computational cost.
• The performance of mSDA is competitive with state-of-the-art methods on benchmark datasets, confirming its practical viability.