[Paper Review] An Introduction to Autoencoders

Umberto Michelucci|arXiv (Cornell University)|Jan 11, 2022

Anomaly Detection Techniques and Applications119 citations

TL;DR

This article explains autoencoders, their architecture (encoder, latent representation, decoder), training objectives, loss functions, and key applications including dimensionality reduction, denoising, anomaly detection, and classification using latent features.

ABSTRACT

In this article, we will look at autoencoders. This article covers the mathematics and the fundamental concepts of autoencoders. We will discuss what they are, what the limitations are, the typical use cases, and we will look at some examples. We will start with a general introduction to autoencoders, and we will discuss the role of the activation function in the output layer and the loss function. We will then discuss what the reconstruction error is. Finally, we will look at typical applications as dimensionality reduction, classification, denoising, and anomaly detection. This paper contains the notes of a PhD-level lecture on autoencoders given in 2021.

Motivation & Objective

• Explain what autoencoders are and why they are useful beyond reconstructing inputs.
• Describe the standard autoencoder architecture and the role of the latent bottleneck.
• Discuss common training losses (MSE and binary cross-entropy) and activation choices.
• Illustrate how autoencoders enable dimensionality reduction and feature extraction for downstream tasks.
• Summarize applications including denoising and anomaly detection, with practical MNIST examples.

Proposed method

• Present the three-part autoencoder structure: encoder, latent representation, and decoder.
• Explain the bottleneck as a dimensionality-reducing constraint to avoid identity replication.
• Discuss regularization techniques to encourage meaningful latent representations.
• Describe training objective as minimizing reconstruction loss across the dataset.
• Detail activation choices for the output layer (ReLU or sigmoid) and corresponding loss functions (MSE or BCE).
• Use MNIST-based examples to illustrate how latent features can perform dimensionality reduction and enable faster classification.

Experimental results

Research questions

• RQ1 What constitutes an autoencoder and how is its latent representation learned?
• RQ2 How do bottlenecks and regularization influence the quality and usefulness of learned representations?
• RQ3 What loss functions and activation choices are appropriate for autoencoders given data normalization?
• RQ4 What practical benefits do latent features provide for dimensionality reduction, classification, and anomaly detection?

Key findings

• Autoencoders learn informative latent representations by reconstructing inputs through an encoder–latent–decoder pipeline.
• A bottleneck forces dimensionality reduction, enabling compact learned features useful for downstream tasks.
• MSE and BCE are common loss functions; BCE requires normalized inputs and sigmoid outputs, while MSE is broadly applicable.
• Latent features can dramatically speed up classification (e.g., kNN) with modest accuracy loss compared to using full inputs.
• Autoencoders can be employed for dimensionality reduction with advantages over PCA in handling non-linear transforms and large datasets via mini-batch training.
• Denoising autoencoders and anomaly detection demonstrate practical uses by reconstructing clean inputs from noisy data and identifying outliers based on reconstruction error.