[Paper Review] Benchmarking Compositionality with Formal Languages

Josef Valvoda, Naomi Saphra+3 more|arXiv (Cornell University)|Aug 17, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper investigates whether neural sequence-to-sequence models can learn compositional string-to-string transformations using formal languages. By sampling from deterministic finite-state transducers (SFSTs), the authors show that models either fully generalize or fail completely, with transition coverage—requiring ~400 examples per transition—emerging as a key predictor of learnability, though exceptions like the SCAN dataset suggest more complex formalisms may be needed for a complete theory.

ABSTRACT

Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability. Whether large neural models in NLP can acquire this ability while learning from data is an open question. In this paper, we investigate this problem from the perspective of formal languages. We use deterministic finite-state transducers to make an unbounded number of datasets with controllable properties governing compositionality. By randomly sampling over many transducers, we explore which of their properties contribute to learnability of a compositional relation by a neural network. We find that the models either learn the relations completely or not at all. The key is transition coverage, setting a soft learnability limit at 400 examples per transition.

Motivation & Objective

• To investigate whether neural sequence-to-sequence models can learn compositional generalization in string-to-string transformations.
• To evaluate the role of formal language properties in determining the learnability of systematic relations by neural networks.
• To identify a reliable, data-driven complexity metric for predicting model generalization behavior.
• To challenge existing benchmarks like SCAN by analyzing their underlying formal structure using finite-state transducers.
• To explore whether higher-level formalisms beyond subregular languages may yield more consistent learnability predictions.

Proposed method

• The authors use deterministic finite-state transducers (SFSTs) to generate a large, diverse set of controllable string-to-string transformation tasks.
• They randomly sample 2,800 SFSTs and generate training datasets by sampling input-output pairs from each transducer.
• Neural sequence-to-sequence models are trained on these datasets to assess their ability to generalize compositionally.
• The key metric for analysis is transition coverage—the number of training examples that traverse each transition in the SFST.
• The study compares model performance across SFSTs with varying structural complexity and coverage levels.
• The SCAN dataset is reverse-engineered as an SFST to test the predictive power of transition coverage in real-world benchmarks.

Experimental results

Research questions

• RQ1 Can neural sequence-to-sequence models achieve perfect compositional generalization on string-to-string transformations generated by finite-state transducers?
• RQ2 What formal language properties of SFSTs most strongly predict whether a neural model will learn a given transformation?
• RQ3 Does transition coverage—defined as the number of training examples per transition—serve as a reliable predictor of model generalization?
• RQ4 Why does the SCAN dataset, despite low transition coverage, remain learnable by neural models?
• RQ5 Can a more nuanced complexity metric, possibly rooted in higher-level formalisms like context-free grammars, better predict model learnability than subregular-level measures?

Key findings

• Neural sequence-to-sequence models exhibit a sharp threshold in generalization: they either fully generalize or fail completely, with no partial success.
• Transition coverage of approximately 400 examples per transition emerges as a strong predictor of successful generalization across randomly sampled SFSTs.
• The SCAN dataset is an exception to the transition coverage rule, remaining learnable despite low coverage, suggesting it may be better described by a more complex formalism.
• When attention mechanisms are added, models reliably learn even complex regular string copying functions, and their attention patterns mirror the derivations of the underlying transducers.
• The results suggest that subregular-level complexity metrics like transition coverage are insufficient for predicting learnability in all cases, especially when higher-level grammatical structure is involved.
• The study implies that future work should explore formalisms higher in the Chomsky hierarchy to identify a native complexity metric that better reflects model learning behavior.