[Paper Review] Autoencoders

Dor Bank, Noam Koenigstein+1 more|arXiv (Cornell University)|Mar 12, 2020

Time Series Analysis and Forecasting123 citations

TL;DR

A comprehensive survey of autoencoder types, regularization methods, variational autoencoders, and their wide range of applications and advanced extensions.

ABSTRACT

An autoencoder is a specific type of a neural network, which is mainly designed to encode the input into a compressed and meaningful representation, and then decode it back such that the reconstructed input is similar as possible to the original one. This chapter surveys the different types of autoencoders that are mainly used today. It also describes various applications and use-cases of autoencoders.

Motivation & Objective

• Define the autoencoder framework and formalize encoder/decoder objectives.
• Explain regularization strategies that prevent learning the identity function and promote meaningful representations.
• Introduce variational autoencoders and their optimization via variational inference.
• Survey common applications of autoencoders across generation, classification, clustering, anomaly detection, and recommender systems.
• Discuss advanced autoencoder techniques and their relation to GANs and other generative models.

Proposed method

• Describe the basic autoencoder objective to minimize reconstruction loss with encoder A and decoder B.
• Discuss bottleneck and regularization to avoid identity mapping and promote meaningful latent representations.
• Present and explain denoising, sparse, and contractive autoencoders as regularization schemes.
• Introduce variational autoencoders with evidence lower bound and reparameterization trick for gradient-based optimization.
• Explain how disentangled autoencoders modify the KL term to encourage factorized latent representations.
• Outline the reparameterization trick and practical training via mini-batch stochastic optimization.

Experimental results

Research questions

• RQ1 How do regularization techniques affect the quality and generalization of autoencoder learned representations?
• RQ2 What are the differences between standard, denoising, contractive, and sparse autoencoders in learning meaningful latent spaces?
• RQ3 How do variational and disentangled autoencoders formulate and optimize probabilistic latent representations?
• RQ4 What are the primary applications of autoencoders across generation, classification, clustering, and anomaly detection?
• RQ5 How can autoencoders be integrated with or contrasted to GAN-based generative models?

Key findings

• Regularization approaches prevent trivial identity mappings and yield informative latent representations.
• Denoising, sparse, and contractive autoencoders provide robustness and robustness-based regularization for learned features.
• Variational autoencoders introduce a probabilistic framework with a lower bound objective and the reparameterization trick for scalable training.
• Disentangled autoencoders leverage a beta factor to encourage less correlated latent features and more interpretable factors.
• Autoencoders are applicable as generative models, as feature extractors for classification or clustering, and for anomaly detection and recommender systems.
• Advanced hybrids with GANs and adversarial training address limitations like blurriness in VAE outputs and mode collapse in GANs.