[Paper Review] Break It Down: Evidence for Structural Compositionality in Neural Networks

Michael A. Lepori, T. Serre+1 more|arXiv (Cornell University)|Jan 26, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces structural compositionality as a property of neural networks, demonstrating through model pruning that many vision and language models implement subroutines via modular, ablatable subnetworks. The key finding is that models often decompose complex tasks into discrete, functional components, suggesting implicit compositionality without symbolic mechanisms.

ABSTRACT

Though modern neural networks have achieved impressive performance in both vision and language tasks, we know little about the functions that they implement. One possibility is that neural networks implicitly break down complex tasks into subroutines, implement modular solutions to these subroutines, and compose them into an overall solution to a task - a property we term structural compositionality. Another possibility is that they may simply learn to match new inputs to learned templates, eliding task decomposition entirely. Here, we leverage model pruning techniques to investigate this question in both vision and language across a variety of architectures, tasks, and pretraining regimens. Our results demonstrate that models often implement solutions to subroutines via modular subnetworks, which can be ablated while maintaining the functionality of other subnetworks. This suggests that neural networks may be able to learn compositionality, obviating the need for specialized symbolic mechanisms.

Motivation & Objective

• To investigate whether modern neural networks implicitly implement compositional solutions through modular subnetworks rather than template matching.
• To determine if structural compositionality emerges in pre-trained models and how pretraining influences it.
• To develop and apply a pruning-based method to detect and analyze modular subnetworks that support specific subroutines.
• To evaluate whether subnetworks for different subroutines can be independently ablated while preserving functionality in others, indicating modularity.

Proposed method

• The authors use continuous sparsification and iterative pruning to identify sparse subnetworks that maintain performance on specific subroutines.
• They define structural compositionality as the presence of subnetworks that achieve high accuracy on one subroutine but low on another, indicating functional modularity.
• Subnetworks are discovered by masking weights and evaluating performance on examples requiring specific subroutines (e.g., 'inside' vs. 'contact' in vision tasks).
• Performance differences between subroutines are measured as the difference in accuracy on examples requiring each, with positive differences indicating compositional structure.
• Subnetwork overlap is quantified using intersection over union (IoU) across layers to assess consistency and modularity.
• The analysis is applied across vision (ResNet50 on Number-Contact) and language models, with comparisons across pretraining regimes.

Figure 1: (Left) An illustration of the tasks used to study structural compositionality. Stimuli are generated via the composition of two subroutines: (+/- Inside) and (+/- Contact) . These stimuli are used to construct odd-one-out tasks, where the model is tasked with identifying the image that doe

Experimental results

Research questions

• RQ1 Do neural networks implement complex tasks through modular, reusable subnetworks that correspond to distinct subroutines?
• RQ2 Can subnetworks be ablated independently such that performance on one subroutine degrades while performance on another remains stable?
• RQ3 To what extent does unsupervised pretraining enhance the structural compositionality of fine-tuned models?
• RQ4 How consistent are subnetworks across multiple runs for the same subroutine, and how do they compare across different subroutines?
• RQ5 Do subnetworks for different subroutines show low overlap, indicating functional separation?

Key findings

• Pruning reveals that many models implement subroutines via modular subnetworks, with performance differences between subroutines indicating functional specialization.
• For the Inside subroutine in vision tasks, subnetworks show high IoU (e.g., 0.974 at backbone.layer4.0.conv1), indicating consistent, modular structure.
• For the Number subroutine, subnetworks show lower IoU (e.g., 0.370 at backbone.layer4.0.conv1), suggesting noisier or less consistent discovery, but still distinct from the Inside task.
• Between-task subnetwork overlap is consistently low (e.g., 0.122 at backbone.layer4.0.conv1), confirming that subnetworks for different subroutines are functionally separate.
• Pruned models retain similar structural compositionality patterns as full models, validating the robustness of the method.
• Unsupervised pretraining leads to more consistently compositional structures in language models, suggesting pretraining supports modular learning.

Figure 2: Illustration of the experimental design. For brevity, we denote “subroutine” as SR in the diagram. (A) First, we train a neural network on a compositional task (Inside-Contact) , ensuring that it can achieve high accuracy on the task. (B) We then optimize a binary mask over weights, such t