[Paper Review] On the exact relationship between the denoising function and the data distribution

Heikki Arponen, Matti Herranen+1 more|arXiv (Cornell University)|Sep 6, 2017

NMR spectroscopy and applications3 references3 citations

TL;DR

This paper establishes an exact, invertible relationship between the optimal denoising function and the data distribution under additive Gaussian noise, showing that the denoising function is mathematically equivalent to the score function of the corrupted data distribution. The key contribution is a generalization of prior results—previously valid only in the small-noise limit—to arbitrary noise levels, proving that denoising implicitly models the full data manifold structure via the gradient of the log-probability density.

ABSTRACT

We prove an exact relationship between the optimal denoising function and the data distribution in the case of additive Gaussian noise, showing that denoising implicitly models the structure of data allowing it to be exploited in the unsupervised learning of representations. This result generalizes a known relationship [2], which is valid only in the limit of small corruption noise.

Motivation & Objective

• To establish a precise, exact mathematical relationship between the optimal denoising function and the underlying data distribution under additive Gaussian noise.
• To generalize prior theoretical results that were only valid in the limit of small corruption noise.
• To demonstrate that denoising learning implicitly captures the full structure of the data manifold by modeling the score function of the corrupted data distribution.
• To show that the denoising function can be inverted to recover the data distribution, enabling exact representation learning.
• To lay a theoretical foundation for using denoising autoencoders as a principled method for unsupervised representation learning.

Proposed method

• Derives the optimal denoising function as the minimum mean square error (MMSE) estimator, given by the conditional expectation $ g^*(\tilde{x}) = \mathbb{E}[x|\tilde{x}] $.
• Uses Bayes' rule to express the conditional expectation in terms of the prior data distribution $ p(x) $ and the corruption likelihood $ p(\tilde{x}|x) $.
• Applies the specific form of additive Gaussian noise $ \tilde{x} = x + \sigma_n \epsilon $, with $ \epsilon \sim \mathcal{N}(0,I) $, to derive the exact relationship.
• Derives the key identity $ x p(\tilde{x}|x) = \tilde{x} p(\tilde{x}|x) + \sigma_n^2 \nabla_{\tilde{x}} p(\tilde{x}|x) $, which links the clean input to the corrupted observation.
• Combines this identity with marginalization over $ x $ to show that $ g^*(\tilde{x}) = \tilde{x} + \sigma_n^2 \nabla_{\tilde{x}} \log p(\tilde{x}) $, where $ p(\tilde{x}) $ is the corrupted data distribution.
• Demonstrates that the relationship is invertible via contour integration, allowing reconstruction of $ p(\tilde{x}) $ from $ g^* $, and thus of $ p(x) $ via deconvolution.

Experimental results

Research questions

• RQ1 What is the exact mathematical relationship between the optimal denoising function and the data distribution under additive Gaussian noise?
• RQ2 How does this relationship generalize prior results that were only valid in the small-noise limit?
• RQ3 Can the denoising function be used to recover the full data distribution, including its manifold structure?
• RQ4 Is the mapping between the denoising function and the data distribution invertible?
• RQ5 What is the theoretical justification for using denoising as a representation learning method in deep learning?

Key findings

• The optimal denoising function is exactly $ g^*(\tilde{x}) = \tilde{x} + \sigma_n^2 \nabla_{\tilde{x}} \log p(\tilde{x}) $, where $ p(\tilde{x}) $ is the corrupted data distribution.
• This result generalizes the known small-noise approximation $ g^*(\tilde{x}) = \tilde{x} + \sigma_n^2 \nabla_{\tilde{x}} \log p_X(\tilde{x}) + o(\sigma_n^2) $ to arbitrary noise levels.
• The relationship between the denoising function and the data distribution is invertible: $ p(\tilde{x}) $ can be recovered from $ g^* $ via a contour integral.
• The denoising function encodes the full structure of the data manifold through the score function of the corrupted distribution.
• The mapping $ p(x) \longleftrightarrow p(\tilde{x}) \longleftrightarrow g^*(\tilde{x}) $ is exact and invertible, proving that denoising learns the full data distribution.
• The result shows that denoising autoencoders can learn complex data structures beyond simple conditional expectations, unlike standard regression.