[Paper Review] Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

Daniel Keysers, Nathanael Schärli+12 more|arXiv (Cornell University)|Dec 20, 2019

Topic Modeling39 references141 citations

TL;DR

The paper formalizes distribution-based compositionality assessment (DBCA), builds the CFQ dataset to measure compositional generalization, and shows standard models underperform as compound divergence between train/test rises, with atom distribution kept similar.

ABSTRACT

State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures. We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy. We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.

Motivation & Objective

• Define a principled method to assess compositional generalization using train/test splits that maximize compound divergence while keeping atom distributions similar.
• Introduce CFQ, a large realistic NLQ→SPARQL dataset designed for compositionality evaluation.
• Provide a framework to construct and compare compositionality splits across datasets (CFQ and SCAN).
• Analyze baseline neural architectures on these splits to quantify their compositional generalization capabilities.

Proposed method

• Introduce Distribution-Based Compositionality Assessment (DBCA) to quantify atom and compound divergences between train and test sets.
• Represent each example as composed of atoms (rules) and compounds (rule applications) and compute divergences using weighted subgraph distributions and Chernoff coefficients (Bhattacharyya for atoms, 0.1-parameter for compounds).
• Construct CFQ via automatic, rule-based generation with an explicit DAG of rule applications to track atoms/compounds.
• Use an iterative greedy algorithm to create train/test splits with target compound divergence and constrained atom divergence (≤0.02).
• Provide comparisons to other compositional splits (e.g., output/input length, pattern-based splits) and analyze across CFQ and SCAN.
• Evaluate three baselines (LSTM+attention, Transformer, Universal Transformer) on CFQ and SCAN under various divergence-based splits.

Experimental results

Research questions

• RQ1 How to quantify the suitability of a split for measuring compositional generalization (DBCA principles)?
• RQ2 What is the impact of maximizing compound divergence while keeping atom divergence low on model performance?
• RQ3 Do state-of-the-art architectures generalize compositionally on realistic benchmarks like CFQ and SCAN?
• RQ4 Can CFQ and the proposed splits reveal robustness gaps in neural models for semantic parsing and navigation tasks?

Key findings

• Baseline architectures (LSTM+attention, Transformer, Universal Transformer) fail to generalize compositionally on CFQ MCD splits (mean accuracy < 20%).
• There is a strong negative correlation between compound divergence and accuracy across all models and tasks.
• CFQ and SCAN splits with maximum compound divergence but low atom divergence yield higher difficulty than random or other traditional splits.
• On CFQ, random splits yield >95% accuracy, but MCD splits drop substantially across all models (e.g., ~14.9–18.9% vs ~97–99%).
• Compound divergence is a strong predictor of test accuracy, more so than simple length-based or pattern-based split criteria.
• CFQ provides richer compositional annotations and more diverse query patterns than prior semantic-parsing datasets, enabling robust compositionality analysis.