[Paper Review] Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks

Yuxuan Li, James L. McClelland|arXiv (Cornell University)|Oct 2, 2022

Topic Modeling4 citations

TL;DR

This paper demonstrates that two-layer causal transformers with label-based order encoding achieve strong systematic generalization on algorithmic tasks like copying, reversing, and hierarchical sorting, generalizing to sequences longer than seen during training. The models develop interpretable, task-decomposed attention patterns and exploit shared computation across related tasks, revealing how self-attention enables structured computation without explicit symbolic components.

ABSTRACT

Transformer networks have seen great success in natural language processing and machine vision, where task objectives such as next word prediction and image classification benefit from nuanced context sensitivity across high-dimensional inputs. However, there is an ongoing debate about how and when transformers can acquire highly structured behavior and achieve systematic generalization. Here, we explore how well a causal transformer can perform a set of algorithmic tasks, including copying, sorting, and hierarchical compositions of these operations. We demonstrate strong generalization to sequences longer than those used in training by replacing the standard positional encoding typically used in transformers with labels arbitrarily paired with items in the sequence. We search for the layer and head configuration sufficient to solve these tasks, then probe for signs of systematic processing in latent representations and attention patterns. We show that two-layer transformers learn reliable solutions to multi-level problems, develop signs of task decomposition, and encode input items in a way that encourages the exploitation of shared computation across related tasks. These results provide key insights into how attention layers support structured computation both within a task and across multiple tasks.

Motivation & Objective

• To investigate how transformers achieve systematic generalization on structured algorithmic tasks without explicit symbolic inductive biases.
• To evaluate the effectiveness of label-based order encoding versus standard positional encodings in enabling length generalization.
• To analyze attention mechanisms and latent representations for signs of task decomposition and shared computation across related tasks.
• To understand the computational dynamics of multi-headed attention in supporting structured, multi-level reasoning.

Proposed method

• Trained a two-layer causal transformer on a dataset of 100k sequences (5–50 items) with disentangled feature representations using onehot/multihot token encodings.
• Replaced standard positional encodings with random labels assigned to each item to communicate sequence order, enabling generalization to longer sequences.
• Used task-specific onehot tokens to condition the model on the desired operation (e.g., copy, sort, group).
• Performed attention ablation and representation probing to analyze head-level and layer-level dynamics in latent space.
• Evaluated model performance on test sequences longer than the maximum training length (50 items).
• Analyzed attention patterns and feature representations to detect signs of systematic processing and task decomposition.

Experimental results

Research questions

• RQ1 Can a minimal two-layer causal transformer learn and systematically generalize across multiple algorithmic tasks involving sequence manipulation?
• RQ2 How does label-based order encoding compare to standard positional encodings in enabling length generalization?
• RQ3 Do attention heads in the transformer develop interpretable, task-specific roles that reflect systematic decomposition of complex operations?
• RQ4 To what extent do latent representations encode shared structural patterns across related tasks?

Key findings

• Label-based order encoding enabled strong length generalization, allowing models to generalize reliably to sequences longer than the maximum training length of 50 items.
• Two-layer transformers learned reliable solutions to multi-level tasks such as hierarchical sorting and grouping, demonstrating systematic generalization.
• Attention heads exhibited signs of task decomposition, with specific heads specializing in subcomponents like feature grouping or ordering.
• Latent representations encoded item features in a way that facilitated shared computation across related tasks, such as sorting by different feature orders.
• The models developed interpretable attention patterns consistent with known task structures, suggesting emergent symbolic-like processing.
• Learnable and sinusoidal positional encodings resulted in poorer length generalization and slower training compared to the label-based method.