[Paper Review] Conservativeness of untied auto-encoders

Daniel Jiwoong Im, Mohamed Ishmael Belghazi+1 more|arXiv (Cornell University)|Jun 25, 2015

Neural Networks and Applications7 references4 citations

TL;DR

This paper establishes necessary and sufficient conditions for untied auto-encoders to define a conservative vector field, enabling the construction of an energy function analogous to unnormalized log-probability. It shows that conservativeness depends on the symmetry of the reconstruction Jacobian and that denoising training enforces this condition locally, allowing untied auto-encoders to extract the conservative component of vector fields even without tied weights.

ABSTRACT

We discuss necessary and sufficient conditions for an auto-encoder to define a conservative vector field, in which case it is associated with an energy function akin to the unnormalized log-probability of the data. We show that the conditions for conservativeness are more general than for encoder and decoder weights to be the same ("tied weights"), and that they also depend on the form of the hidden unit activation function, but that contractive training criteria, such as denoising, will enforce these conditions locally. Based on these observations, we show how we can use auto-encoders to extract the conservative component of a vector field.

Motivation & Objective

• To determine the necessary and sufficient conditions under which an untied auto-encoder defines a conservative vector field.
• To clarify whether tied weights are necessary for an auto-encoder to have an associated energy function, especially in the untied case.
• To investigate why encoder and decoder weights often become similar during training, despite not being tied.
• To develop a method for extracting the conservative part of a vector field using auto-encoders.
• To extend the theoretical link between auto-encoders and probabilistic models beyond tied-weight architectures.

Proposed method

• The paper derives that an auto-encoder is conservative if and only if its reconstruction Jacobian ∂r(x)/∂x is symmetric over a simply connected domain K.
• It uses differential forms and exterior calculus to show that the 1-form associated with the auto-encoder’s vector field is exact (i.e., conservative) if and only if the Jacobian is symmetric.
• The derivation relies on Poincaré’s theorem, which states that a closed form (dα = 0) is exact in a simply connected domain.
• The condition dα = 0 is shown to be equivalent to the symmetry of the Jacobian matrix ∂r_i/∂x_j = ∂r_j/∂x_i for all i, j.
• The method enables the recovery of an energy function F(x) via integration of the conservative vector field r(x) − x = ∇F(x), even when weights are untied.
• The approach is validated experimentally on MNIST, showing that denoising auto-encoders learn increasingly conservative fields near training data.

Experimental results

Research questions

• RQ1 Under what conditions does an untied auto-encoder define a conservative vector field?
• RQ2 Is tied weight initialization necessary for an auto-encoder to have an associated energy function?
• RQ3 Why do encoder and decoder weights in untied auto-encoders often become similar during training?
• RQ4 Can the conservative component of a general vector field be extracted using auto-encoders?
• RQ5 How does denoising training enforce conservativeness locally near the data manifold?

Key findings

• The necessary and sufficient condition for an auto-encoder to be conservative is that its reconstruction Jacobian is symmetric for all x in a simply connected domain.
• Tied weights are sufficient but not necessary for conservativeness; the key factor is Jacobian symmetry, which can emerge even without weight tying.
• Contractive training criteria such as denoising enforce Jacobian symmetry locally near the data manifold, promoting conservativeness.
• The paper demonstrates that untied auto-encoders can extract the conservative part of a vector field, as shown by the increasing favoring of original MNIST data points over corrupted ones in the energy function.
• On average, the potential energy of the conservative vector field trained by untied auto-encoders favors original data points over corrupted ones, with values rising from 0.5072 at F=0 to 0.9968 at F=K.
• The energy function of a tied auto-encoder with sigmoid units is mathematically equivalent to the free energy of a Gaussian RBM, confirming a deep link between these models.