[Paper Review] Flexible Compositional Learning of Structured Visual Concepts

Yanli Zhou, Brenden M. Lake|arXiv (Cornell University)|May 20, 2021

Multimodal Machine Learning Applications22 references4 citations

TL;DR

This paper proposes a Bayesian program induction model that flexibly learns structured visual concepts through compositional generalization from few examples, using a probabilistic language of thought to represent diverse relational structures. The model closely fits human behavioral data across multiple composition types—fixed configurations, free part combinations, orientation constraints, and paired elements—demonstrating strong inductive biases for orientation invariance and flexible configuration reuse.

ABSTRACT

Humans are highly efficient learners, with the ability to grasp the meaning of a new concept from just a few examples. Unlike popular computer vision systems, humans can flexibly leverage the compositional structure of the visual world, understanding new concepts as combinations of existing concepts. In the current paper, we study how people learn different types of visual compositions, using abstract visual forms with rich relational structure. We find that people can make meaningful compositional generalizations from just a few examples in a variety of scenarios, and we develop a Bayesian program induction model that provides a close fit to the behavioral data. Unlike past work examining special cases of compositionality, our work shows how a single computational approach can account for many distinct types of compositional generalization.

Motivation & Objective

• To investigate how humans generalize across diverse types of visual composition in few-shot learning scenarios.
• To develop a unified computational model that accounts for multiple forms of compositional generalization, including fixed configurations, free part combinations, orientation constraints, and paired elements.
• To evaluate whether a single Bayesian program induction framework can capture the richness of human inductive biases in visual concept learning.

Proposed method

• The study uses a probabilistic language of thought (PLoT) to define a space of generative programs for abstract alien figures, encoding parts, relations, and spatial configurations.
• A Bayesian inference framework scores programs based on their posterior probability given observed examples, favoring those that best explain the data under a prior over compositional structures.
• The model employs two key grammar operations: 'attach' for generating all possible configurations of two parts, and 'attach*' for specifying a single, target configuration.
• The model is fitted to behavioral data using maximum a posteriori (MAP) estimation, with free parameters capturing human inductive biases such as orientation invariance and configuration flexibility.
• The framework supports generative sampling of new concepts, enabling comparison with human-generated examples and model evaluation on generative tasks.
• Alternative models—such as string-based models and pre-trained CNNs—are used for comparison to isolate the role of compositional structure and relational reasoning.

Experimental results

Research questions

• RQ1 Can humans make meaningful compositional generalizations across diverse visual concept types (e.g., fixed configurations, free combinations, orientation constraints, paired elements) from just a few examples?
• RQ2 How do human inductive biases—such as orientation invariance and tolerance for novel configurations—shape their concept learning behavior in few-shot settings?
• RQ3 To what extent can a single Bayesian program induction model account for multiple types of compositional generalization in visual learning?

Key findings

• The Bayesian program induction model provides a close fit to human generalization patterns across all tested concept types, significantly outperforming baseline models like the string-GCM and pre-trained CNNs.
• Participants exhibited a strong preference for orientation-invariant concepts, reflected in a high MAP estimate for the probability of orientation invariance in the model’s grammar.
• Humans demonstrated flexibility in generalizing beyond observed configurations, indicating a bias toward reusable, abstract relational rules rather than rigid memorization of specific layouts.
• The model’s ability to generalize to novel configurations and unseen primitives highlights its capacity to capture higher-level compositional abstractions, such as part repetition and relational constraints.
• The fitted model parameters revealed psychologically meaningful inductive biases, including a preference for concepts that do not require fixed configurations, suggesting a strong bias toward compositional generalization.
• The framework enables direct comparison between human and model behavior on both discriminative and generative tasks, offering a path to refine neural network models with human-like inductive biases.