[Paper Review] Towards a Definition of Disentangled Representations

Irina Higgins, David Amos+5 more|arXiv (Cornell University)|Dec 5, 2018

Generative Adversarial Networks and Image Synthesis5 references294 citations

TL;DR

The paper defines disentangled representations formally using group theory and symmetry: a representation is disentangled if it decomposes into subspaces each independently transformed by a subgroup of the world’s symmetry group.

ABSTRACT

How can intelligent agents solve a diverse set of tasks in a data-efficient manner? The disentangled representation learning approach posits that such an agent would benefit from separating out (disentangling) the underlying structure of the world into disjoint parts of its representation. However, there is no generally agreed-upon definition of disentangling, not least because it is unclear how to formalise the notion of world structure beyond toy datasets with a known ground truth generative process. Here we propose that a principled solution to characterising disentangled representations can be found by focusing on the transformation properties of the world. In particular, we suggest that those transformations that change only some properties of the underlying world state, while leaving all other properties invariant, are what gives exploitable structure to any kind of data. Similar ideas have already been successfully applied in physics, where the study of symmetry transformations has revolutionised the understanding of the world structure. By connecting symmetry transformations to vector representations using the formalism of group and representation theory we arrive at the first formal definition of disentangled representations. Our new definition is in agreement with many of the current intuitions about disentangling, while also providing principled resolutions to a number of previous points of contention. While this work focuses on formally defining disentangling - as opposed to solving the learning problem - we believe that the shift in perspective to studying data transformations can stimulate the development of better representation learning algorithms.

Motivation & Objective

• Motivate a principled, formal definition of disentangled representations using symmetry transformations.
• Bridge concepts from physics (group and representation theory) to machine learning representations.
• Clarify what constitutes data generative factors and how they may be represented and manipulated.

Proposed method

• Introduce symmetry transformations as group actions that change only some aspects of the world while leaving others invariant.
• Propose that a vector representation is disentangled if it decomposes into independent subspaces each affected by a single subgroup of the world’s symmetry group.
• Define equivariance: a representation f is disentangled if there exists a G-action on Z such that f is G-equivariant with respect to the world W.
• Formalize disentangled representations with respect to a decomposition G = G1 × ... × Gn and a corresponding decomposition of Z into Z1 ⊕ ... ⊕ Zn or Z1 × ... × Zn.
• Discuss linear disentangled representations where subgroup actions on their subspaces are linear.
• Provide a worked grid-world example to illustrate the concepts.

Experimental results

Research questions

• RQ1 How can symmetry transformations be formalized to define disentangled representations?
• RQ2 What conditions on a world’s symmetry group decomposition ensure a representation that separates factors of variation into independent subspaces?
• RQ3 How does equivariance relate to the ability to map world states to a representation space while preserving symmetry structure?
• RQ4 What are the implications of choosing different subgroup decompositions for a given dataset?

Key findings

• Propose the first principled, formal definition of disentangled representations based on group and representation theory.
• Show that a disentangled representation corresponds to a decomposition of the representation space aligned with a decomposition of the world’s symmetry group.
• Argue that multiple subgroup decompositions may exist, but only natural decompositions reflecting world structure yield useful disentangling.
• Highlight that disentangled representations can enable compositionality and potential linearity in transformations, aiding learning efficiency.
• Clarify how to evaluate disentangling against the backdrop of symmetry-based definitions rather than purely empirical intuitions.