[Paper Review] Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing

Linlu Qiu, Peter Shaw+6 more|arXiv (Cornell University)|May 24, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper investigates how scaling model size affects compositional generalization in semantic parsing using encoder-decoder (T5) and decoder-only (PaLM) models up to 540B parameters. It compares fine-tuning, prompt tuning, and in-context learning, finding that fine-tuning shows flat or negative scaling curves, while prompt tuning exhibits more positive scaling and outperforms fine-tuning at scale, suggesting parameter-efficient methods may unlock better generalization from scale.

ABSTRACT

Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution compositional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling. Can scaling up model size also improve compositional generalization in semantic parsing? We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for applying a pre-trained language model to a new task: fine-tuning all parameters, prompt tuning, and in-context learning. We observe that fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional generalization in semantic parsing evaluations. In-context learning has positive scaling curves, but is generally outperformed by much smaller fine-tuned models. Prompt-tuning can outperform fine-tuning, suggesting further potential improvements from scaling as it exhibits a more positive scaling curve. Additionally, we identify several error trends that vary with model scale. For example, larger models are generally better at modeling the syntax of the output space, but are also more prone to certain types of overfitting. Overall, our study highlights limitations of current techniques for effectively leveraging model scale for compositional generalization, while our analysis also suggests promising directions for future work.

Motivation & Objective

• To investigate whether increasing model size improves compositional generalization in semantic parsing.
• To compare the scaling behavior of different task adaptation techniques—fine-tuning, prompt tuning, and in-context learning—on compositional generalization.
• To identify error trends and failure modes that vary with model scale and adaptation method.
• To evaluate the impact of distribution shift, output representation, and prompt retrieval on scaling performance.

Proposed method

• Evaluates encoder-decoder T5 models up to 11B parameters and decoder-only PaLM models up to 540B parameters for semantic parsing.
• Applies three task adaptation methods: full fine-tuning, prompt tuning (with learnable soft prompts), and in-context learning using few-shot demonstrations.
• Uses standard benchmarks: GeoQuery and SMCalFlow-CS for compositional generalization evaluation.
• Measures performance across model sizes on out-of-distribution examples involving novel combinations of training elements.
• Analyzes error types, including syntax errors (e.g., unbalanced parentheses), and distributional errors (e.g., using unseen predicates).
• Employs both random and oracle-based prompt retrieval for in-context learning to assess prompt quality effects.

Experimental results

Research questions

• RQ1 Does increasing model size improve compositional generalization in semantic parsing?
• RQ2 How does the scaling behavior of compositional generalization differ across fine-tuning, prompt tuning, and in-context learning?
• RQ3 Are there error patterns in predictions that systematically vary with model scale or adaptation method?
• RQ4 How do distribution shift, output representation, and prompt retrieval quality affect scaling curves?

Key findings

• Fine-tuning exhibits flat or negative scaling curves for compositional generalization, indicating no improvement or even degradation with increased model size.
• Prompt tuning outperforms standard fine-tuning on larger models and shows a more positive scaling curve, suggesting it is better suited for leveraging scale.
• In-context learning shows positive scaling curves but is consistently outperformed by smaller fine-tuned models, indicating limited effectiveness at scale.
• Larger models are better at modeling syntactic structure (e.g., balanced parentheses) but are more prone to overfitting on certain compositional patterns.
• Error analysis reveals that larger models are more likely to generate outputs that exactly match training examples, even when incorrect, suggesting memorization or distributional bias.
• The use of an oracle prompt retriever improves in-context learning performance but still fails to match smaller fine-tuned models, highlighting limitations in prompt engineering.