[Paper Review] Compositional Generalization from First Principles

Thaddäus Wiedemer, P. Mayilvahanan+2 more|arXiv (Cornell University)|Jul 10, 2023

Geochemistry and Geologic Mapping4 citations

TL;DR

This paper proposes a formal framework for compositional generalization in machine learning by modeling data generation as a compositional process with identifiable components. It derives sufficient conditions on the support of the training distribution and model architecture that guarantee generalization to unseen compositions, validated through synthetic experiments and theoretical analysis, offering a principled foundation for scalable, sample-efficient learning.

ABSTRACT

Leveraging the compositional nature of our world to expedite learning and facilitate generalization is a hallmark of human perception. In machine learning, on the other hand, achieving compositional generalization has proven to be an elusive goal, even for models with explicit compositional priors. To get a better handle on compositional generalization, we here approach it from the bottom up: Inspired by identifiable representation learning, we investigate compositionality as a property of the data-generating process rather than the data itself. This reformulation enables us to derive mild conditions on only the support of the training distribution and the model architecture, which are sufficient for compositional generalization. We further demonstrate how our theoretical framework applies to real-world scenarios and validate our findings empirically. Our results set the stage for a principled theoretical study of compositional generalization.

Motivation & Objective

• To address the persistent failure of machine learning models to generalize compositionally despite compositional data structures.
• To formalize compositionality not as a property of data, but as a property of the data-generating process.
• To derive minimal, sufficient conditions on the support of the training distribution and model architecture for compositional generalization.
• To validate the theoretical framework empirically in synthetic environments with ablation studies.
• To provide a principled foundation for future development of identifiable representation learning with provable generalization.

Proposed method

• The authors define a class of compositional data-generating processes where data is formed by combining known components through a known composition function.
• They model the data generation as a function of latent factors (e.g., position, shape, color) and assume the composition function is known and identifiable.
• Theoretical analysis derives sufficient conditions on the support of the training distribution and model architecture that ensure generalization to novel compositions.
• The framework assumes perfect reconstruction on the training distribution, aligning with identifiable representation learning principles.
• Empirical validation is conducted on synthetic image datasets with two sprites, varying in position, shape, size, and color, to test generalization to unseen compositions.
• Ablation studies examine the necessity of key assumptions, such as full support coverage and known composition functions.

Figure 1: Left : We train a model $f$ to reconstruct images containing two sprites given their latent representation (x, y, shape, size, color) . Center : In the training set (top row and left column), one sprite is fixed in its base configuration (orange triangle or red circle), while the other can

Experimental results

Research questions

• RQ1 Under what conditions on the training distribution and model architecture can a model generalize compositionally to unseen combinations of known components?
• RQ2 Can compositional generalization be guaranteed without relying on inductive biases or explicit architectural priors?
• RQ3 How does the assumption of a known composition function affect the feasibility of provable generalization?
• RQ4 What role does the support of the training distribution play in enabling generalization to novel compositions?
• RQ5 To what extent do empirical methods relying on compositional priors fail to generalize due to insufficient distributional coverage?

Key findings

• Theoretical analysis shows that if the training distribution covers all possible combinations of component factors and the model architecture is sufficiently expressive, compositional generalization is guaranteed.
• Empirical results confirm that models fail to generalize to novel compositions even when components are seen in isolation, highlighting the importance of distributional support.
• Ablation studies demonstrate that the absence of full support coverage leads to systematic failure in generalization, even with perfect training fit.
• The framework reveals that known composition functions are a critical enabling condition for provable generalization, suggesting that unknown or unstructured compositionality undermines generalization.
• The results suggest that current empirical methods may fail to generalize not due to architectural limitations, but due to insufficient coverage of the compositional space in training data.

Figure 2: Compositional representation of a function (Definition 1 ). Component functions $\boldsymbol{\varphi}_{k}$ map each component latent $\boldsymbol{z}_{k}$ to an intermediate representation $\boldsymbol{\tilde{x}}_{k}$ . The composition function $\boldsymbol{C}$ composes them into a final da