[Paper Review] Recent Advances in Autoencoder-Based Representation Learning

Michael Tschannen, Olivier Bachem+1 more|arXiv (Cornell University)|Dec 12, 2018

Domain Adaptation and Few-Shot Learning78 references359 citations

TL;DR

An in-depth survey of autoencoder-based representation learning, detailing three main mechanisms to enforce meta-priors (regularizing posteriors, factorizing encoding/decoding, and structured priors) and their relation to supervision and rate-distortion tradeoffs.

ABSTRACT

Learning useful representations with little or no supervision is a key challenge in artificial intelligence. We provide an in-depth review of recent advances in representation learning with a focus on autoencoder-based models. To organize these results we make use of meta-priors believed useful for downstream tasks, such as disentanglement and hierarchical organization of features. In particular, we uncover three main mechanisms to enforce such properties, namely (i) regularizing the (approximate or aggregate) posterior distribution, (ii) factorizing the encoding and decoding distribution, or (iii) introducing a structured prior distribution. While there are some promising results, implicit or explicit supervision remains a key enabler and all current methods use strong inductive biases and modeling assumptions. Finally, we provide an analysis of autoencoder-based representation learning through the lens of rate-distortion theory and identify a clear tradeoff between the amount of prior knowledge available about the downstream tasks, and how useful the representation is for this task.

Motivation & Objective

• Summarize meta-priors guiding representation learning and how autoencoder-based models enforce them.
• Categorize methods by the three major mechanisms: posterior regularization, distribution factorization, and structured priors.
• Relate these methods to supervision levels and practical modeling biases.
• Provide a rate-distortion perspective to understand tradeoffs in unsupervised representations.

Proposed method

• Define variational autoencoder (VAE) framework and ELBO objective.
• Classify mechanisms into: (i) regularizing the encoding/aggregate posterior; (ii) factoring encoding/decoding distributions; (iii) using flexible priors.
• Describe regularizers used on q_phi(z|x) and q_phi(z) (e.g., TC, MMD, HSIC) and their estimators (density-ratio trick, MMD).
• Discuss structured encoding/decoding distributions (deterministic vs stochastic, hierarchical encodings).
• Discuss structured priors (mixtures, hierarchical priors) and how they promote clustering or disentanglement.
• Contrast supervised vs unsupervised setups and explain the information bottleneck viewpoint and rate-distortion framing.

Experimental results

Research questions

• RQ1 What mechanisms are most effective at enforcing disentanglement, hierarchy, and clustering in autoencoder-based representations?
• RQ2 How do regularization of posteriors, distribution factorization, and priors relate to the meta-priors proposed by Bengio et al.?
• RQ3 What is the role of supervision in achieving useful representations for downstream tasks?
• RQ4 How does rate-distortion theory illuminate tradeoffs between prior knowledge and representation usefulness?

Key findings

• Three core mechanisms are identified for enforcing meta-priors: posterior regularization, encoding/decoding distribution structuring, and flexible priors.
• Regularizers such as TC, MMD, HSIC, and information bottleneck-inspired terms help encourage disentanglement and independence.
• Structured encodings (hierarchical or grouped latents) enable modeling of hierarchical or clustered factors of variation.
• Supervision and implicit supervision remain key enablers, with strong inductive biases shaping learned representations.
• A rate-distortion lens reveals a tradeoff between amount of prior knowledge about downstream tasks and the usefulness of the learned representation for those tasks.