[Paper Review] Improving Content-Invariance in Gated Autoencoders for 2D and 3D Object Rotation

Stefan Lattner, Maarten Grachten|arXiv (Cornell University)|Jul 5, 2017

Advanced Vision and Imaging19 references3 citations

TL;DR

This paper proposes a content-invariance regularization (CIR) loss for Gated Autoencoders (GAEs) that improves their ability to learn content-invariant representations for 2D and 3D object rotations. By penalizing cross-reconstruction error between pairs with similar transformation types using a bootstrapped, unsupervised approach, CIR enhances mapping code invariance without compromising reconstruction performance, leading to more robust analogy-making and improved clustering in the latent space.

ABSTRACT

Content-invariance in mapping codes learned by GAEs is a useful feature for various relation learning tasks. In this paper we show that the content-invariance of mapping codes for images of 2D and 3D rotated objects can be substantially improved by extending the standard GAE loss (symmetric reconstruction error) with a regularization term that penalizes the symmetric cross-reconstruction error. This error term involves reconstruction of pairs with mapping codes obtained from other pairs exhibiting similar transformations. Although this would principally require knowledge of the transformations exhibited by training pairs, our experiments show that a bootstrapping approach can sidestep this issue, and that the regularization term can effectively be used in an unsupervised setting.

Motivation & Objective

• To improve content-invariance in GAEs for 2D and 3D object rotation tasks, where existing models show only weak invariance.
• To develop a regularization method that enhances mapping code invariance without requiring labeled transformation types.
• To evaluate whether content-invariance regularization improves both representation quality and downstream tasks like analogy-making.
• To demonstrate that the regularization can be applied in a fully unsupervised setting using a bootstrapping strategy.
• To show that the regularization does not degrade standard GAE reconstruction performance and may even improve it.

Proposed method

• Introduces a content-invariance regularization (CIR) term that penalizes symmetric cross-reconstruction error between image pairs sharing similar transformation types.
• Uses a bootstrapping approach to estimate transformation similarity during training, avoiding the need for ground-truth transformation labels.
• Extends the standard symmetric reconstruction loss of GAEs with the CIR term to jointly optimize for reconstruction and content-invariance.
• Applies the CIR term only to pairs with similar mapping codes, assuming that similar codes indicate similar transformations.
• Gradually increases the influence of the CIR term during training to stabilize learning in the unsupervised setting.
• Employs a GAE architecture with multiplicative (gating) connections to disentangle content and transformation representations.

Experimental results

Research questions

• RQ1 Can a regularization term improve the content-invariance of mapping codes in GAEs for 2D and 3D object rotations?
• RQ2 Is it possible to implement such regularization in an unsupervised setting without ground-truth transformation labels?
• RQ3 Does the CIR regularization improve both representation quality and downstream tasks like analogy-making?
• RQ4 Does the CIR regularization interfere with or improve the standard GAE reconstruction loss?
• RQ5 Can a bootstrapping strategy effectively estimate transformation similarity for regularization without supervision?

Key findings

• The CIR regularization significantly reduces cross-reconstruction error across similar transformation pairs, with the lowest error observed in the GAE+CIR model on the NORB dataset (0.174 vs. 2.351 for standard GAE).
• On the MNISTr20 dataset, the CIR model achieved a symmetric reconstruction error of 0.0130, compared to 0.0173 for the standard GAE, indicating improved reconstruction performance.
• The CIR model showed superior cluster separation in the mapping code space, with higher inter-cluster distances and lower intra-cluster variance, indicating better content-invariance.
• In k-nearest neighbor classification using mapping codes, the CIR model achieved higher accuracy, demonstrating more discriminative and invariant representations.
• Qualitative analogy-making results showed that GAE+CIR produced cleaner, more accurate transformations, especially in ambiguous cases like 180-degree rotations.
• Despite competing objectives, the CIR regularization often reduced the standard GAE loss, suggesting a synergistic effect between reconstruction and invariance.