[Paper Review] On denoising autoencoders trained to minimise binary cross-entropy

Antonia Creswell, Kai Arulkumaran+1 more|arXiv (Cornell University)|Aug 28, 2017

Generative Adversarial Networks and Image Synthesis20 references63 citations

TL;DR

This paper shows that denoising autoencoders (DAEs) trained with binary cross-entropy can approximate the gradient of the data density, enabling gradient-based data space moves toward regions of higher likelihood and enabling novel sample synthesis from noise and improvement of samples from other generative models.

ABSTRACT

Denoising autoencoders (DAEs) are powerful deep learning models used for feature extraction, data generation and network pre-training. DAEs consist of an encoder and decoder which may be trained simultaneously to minimise a loss (function) between an input and the reconstruction of a corrupted version of the input. There are two common loss functions used for training autoencoders, these include the mean-squared error (MSE) and the binary cross-entropy (BCE). When training autoencoders on image data a natural choice of loss function is BCE, since pixel values may be normalised to take values in [0,1] and the decoder model may be designed to generate samples that take values in (0,1). We show theoretically that DAEs trained to minimise BCE may be used to take gradient steps in the data space towards regions of high probability under the data-generating distribution. Previously this had only been shown for DAEs trained using MSE. As a consequence of the theory, iterative application of a trained DAE moves a data sample from regions of low probability to regions of higher probability under the data-generating distribution. Firstly, we validate the theory by showing that novel data samples, consistent with the training data, may be synthesised when the initial data samples are random noise. Secondly, we motivate the theory by showing that initial data samples synthesised via other methods may be improved via iterative application of a trained DAE to those initial samples.

Motivation & Objective

• Motivate and extend the theory that DAEs trained with reconstruction loss relate to the data-generating distribution.
• Show that BCE-based DAEs approximate the gradient of log p(x) and induce data-space gradient ascent.
• Demonstrate practical sampling from random noise and improvement of samples from generative autoencoders.
• Apply the theory to enhance samples from variational and adversarial autoencoders using the denoising criterion.

Proposed method

• Derive the BCE-based reconstruction objective for denoising autoencoders and derive the optimal reconstruction function under BCE (leading to a gradient in data space).
• Prove that the BCE-trained DAE approximates the gradient of the log-density in the limit, analogous to MSE-trained DAEs.
• Train denoising variants of VAE and AAE on CelebA data and use additive Gaussian corruption.
• Demonstrate iterative sampling from noise to data-like samples using the trained reconstruction function.
• Show that iterative application can improve samples generated by DVAE and DAAE.

Experimental results

Research questions

• RQ1 Can BCE-trained DAEs approximate the gradient of the data-generating distribution with respect to the data?
• RQ2 Does iterative application of a BCE-trained denoising autoencoder move samples toward higher-probability regions of the data distribution?
• RQ3 Can BCE-based DAEs improve sample quality for generative models such as DVAE and DAAE?
• RQ4 Is it possible to synthesize novel data samples consistent with the training data starting from random noise using BCE-DAEs?
• RQ5 What practical guidance or techniques (e.g., noise augmentation) aid sampling from such models?

Key findings

• The BCE loss yields a reconstruction function whose limit corresponds to a gradient ascent step toward higher data likelihood.
• Iterative application of a trained BCE-DAE from random noise can generate novel samples consistent with the training data.
• The sampling method can improve initial samples drawn from DVAE and DAAE models.
• Adding small noise between iterations can smooth the data space and help transitions toward higher-density regions.
• The approach provides a pathway to improve samples without retraining, by applying the trained BCE-DAE as a generative operator.