[Paper Review] Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures

Daniel Furrer, Marc van Zee+2 more|arXiv (Cornell University)|Jul 17, 2020

Topic ModelingComputer Science63 references74 citations

TL;DR

The paper compares pre-training and specialized architectures for compositional generalization in semantic parsing on SCAN and CFQ, showing MLM pre-training rivals SCAN-inspired architectures on primitive holds and combined pre-training with intermediate representations yielding state-of-the-art CFQ results.

ABSTRACT

While mainstream machine learning methods are known to have limited ability to compositionally generalize, new architectures and techniques continue to be proposed to address this limitation. We investigate state-of-the-art techniques and architectures in order to assess their effectiveness in improving compositional generalization in semantic parsing tasks based on the SCAN and CFQ datasets. We show that masked language model (MLM) pre-training rivals SCAN-inspired architectures on primitive holdout splits. On a more complex compositional task, we show that pre-training leads to significant improvements in performance vs. comparable non-pre-trained models, whereas architectures proposed to encourage compositional generalization on SCAN or in the area of algorithm learning fail to lead to significant improvements. We establish a new state of the art on the CFQ compositional generalization benchmark using MLM pre-training together with an intermediate representation.

Motivation & Objective

• Survey a range of architectures and techniques previously applied to SCAN and CFQ for compositional generalization.
• Evaluate two representative CGP architectures on DBCA-based splits to assess transferability across tasks.
• Assess MLM pre-training effectiveness on SCAN and CFQ and its interaction with intermediate representations.
• Identify whether SCAN-inspired or algorithm-learning approaches provide robust gains beyond SCAN/CFQ-specific splits.
• Establish a new state-of-the-art CFQ performance by combining pre-training with an intermediate representation.

Proposed method

• Review and categorize architectures and techniques for compositional generalization (general seq2seq, SCAN-inspired, algorithm learning, pre-training, intermediate representations).
• Evaluate architectures on DBCA-based MCD splits for SCAN and CFQ to measure compositional generalization under distributional shifts.
• Fine-tune and compare MLM pre-trained models (T5 variants) against non-pre-trained baselines on SCAN and CFQ splits.
• Introduce an intermediate SPARQL representation for CFQ to align NL questions with output structure and test its impact with T5-11B.
• Report and analyze results across multiple splits, including MCD-mean, and provide state-of-the-art CFQ performance.

Experimental results

Research questions

• RQ1 Does MLM pre-training improve compositional generalization on SCAN and CFQ compared to SCAN-inspired architectures?
• RQ2 Are SCAN-inspired architectures (CGPS, Neural Shuffle Exchange Network) robust to distributional shifts measured by DBCA/MCD splits?
• RQ3 Does combining pre-training with an intermediate representation yield superior CFQ performance?
• RQ4 How do general-purpose seq2seq architectures compare to specialized CG models on compositional generalization benchmarks?
• RQ5 What insights arise about evaluating compositional generalization beyond traditional SCAN splits?

Key findings

• MLM pre-training yields significant gains on CFQ and primitive-holdout SCAN splits relative to non-pre-trained baselines.
• Pre-training provides substantial improvement on CFQ but can reduce performance on the SCAN length split, suggesting limited length generalization benefits.
• SCAN-inspired CG mechanisms (CGPS, NSEN) show strong results on some SCAN splits but fail to consistently outperform general-purpose architectures on CFQ or across DBCA splits.
• General-purpose architectures (Transformer, LSTM+A, Evolved Transformer) deliver robust gains, with no clear, consistent superiority of specialized architectures across all splits.
• Combining MLM pre-training with an intermediate SPARQL representation yields a new state-of-the-art CFQ score (42.1% on the MCD-mean split).
• Results indicate that improvements on SCAN do not automatically transfer to CFQ, and DBCA-based evaluation reveals variability across architectures.