[Paper Review] CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing

Andy Rosenbaum, Saleh Soltan+4 more|arXiv (Cornell University)|Oct 13, 2022

Natural Language Processing Techniques4 citations

TL;DR

CLASP proposes a few-shot cross-lingual data augmentation method that uses a large language model (AlexaTM 20B) to generate synthetic semantic parsing training data for low-resource settings. By prompting the LLM to generate both text and parses in the same or target languages—using slot replacement, translation, or joint generation—CLASP improves zero-shot cross-lingual generalization, achieving 6.1 absolute improvement on mTOP and 4.79 points on English Pizza with only 16 training examples.

ABSTRACT

A bottleneck to developing Semantic Parsing (SP) models is the need for a large volume of human-labeled training data. Given the complexity and cost of human annotation for SP, labeled data is often scarce, particularly in multilingual settings. Large Language Models (LLMs) excel at SP given only a few examples, however LLMs are unsuitable for runtime systems which require low latency. In this work, we propose CLASP, a simple method to improve low-resource SP for moderate-sized models: we generate synthetic data from AlexaTM 20B to augment the training set for a model 40x smaller (500M parameters). We evaluate on two datasets in low-resource settings: English PIZZA, containing either 348 or 16 real examples, and mTOP cross-lingual zero-shot, where training data is available only in English, and the model must generalize to four new languages. On both datasets, we show significant improvements over strong baseline methods.

Motivation & Objective

• To address the scarcity of human-annotated training data in semantic parsing, especially in multilingual settings.
• To overcome the limitations of standard data augmentation in semantic parsing, where text modifications often break parse consistency.
• To enable effective few-shot cross-lingual generalization by generating high-quality synthetic training examples using a large language model.
• To improve performance of moderate-sized models (e.g., 500M parameters) in low-resource settings without relying on large-scale pretraining or runtime LLMs.
• To develop a robust, end-to-end data augmentation pipeline that maintains text-parse alignment across languages and handles failure modes like slot misalignment or casing errors.

Proposed method

• CLASP uses four prompting strategies: RS (replace slots and generate text in the same language), TS (translate slots and generate text in a new language), GB (generate both parse and text in the same language), and TB (translate both parse and text in a cross-lingual setting).
• For each method, the LLM is prompted with a few context examples (verbalized as 'Semantic Parse: ... Translation in English: ...') to ensure consistent output format and improve zero-shot generalization.
• The method includes a 'Fix Casing' recovery mechanism that corrects case mismatches in slot values (e.g., 'nicole' → 'Nicole') by finding case-insensitive matches in the generated text.
• For cross-lingual generation, CLASP uses machine translation (Opus or LLM-based) followed by Sim-Align to align translated text to the original parse structure.
• Outputs are filtered using four failure mode heuristics: missing slot value, discontiguous target, verbatim copy of input, and inclusion of the word 'Sentence' in the output.
• The final synthetic dataset is used to fine-tune a smaller model (e.g., 500M parameters) to improve performance in low-resource and zero-shot cross-lingual settings.

Figura 1: Cross-lingual Data Augmentation: AlexaTM 20B sees only a few examples of mTOP Semantic Parsing and can generate data in multiple languages.

Experimental results

Research questions

• RQ1 Can LLM-based data augmentation significantly improve few-shot semantic parsing performance in low-resource settings?
• RQ2 How effective is cross-lingual data generation via LLM prompting in maintaining text-parse alignment across languages?
• RQ3 What are the dominant failure modes in machine translation-based semantic parsing, and can they be mitigated with filtering and recovery mechanisms?
• RQ4 Does joint generation of text and parse (GB/TB) outperform sequential generation (RS/TS) in preserving semantic consistency?
• RQ5 Can synthetic data generated from a 20B-parameter LLM effectively boost the performance of a 500M-parameter model on zero-shot cross-lingual semantic parsing?

Key findings

• On the English Pizza dataset with only 16 training examples, CLASP improved Unordered Exact Match from 80.40 to 85.19, a gain of 4.79 points.
• On the mTOP cross-lingual zero-shot benchmark, CLASP achieved a 6.1-point improvement in Exact Match (from 60.3 to 66.4) compared to machine translation with slot alignment.
• The success rate of filtered machine translation outputs varied from 58.3% (Hindi) to 86.7% (German), indicating significant challenges in cross-lingual alignment, especially for distant language pairs.
• The 'Fix Casing' recovery mechanism successfully corrected case mismatches in 13.4% of 20B-generated outputs that otherwise failed due to casing issues.
• Filtering out outputs with 'Sentence' in the text (13.4% of Hindi outputs) was critical, suggesting that prompt design and output format constraints are essential for reliable generation.
• CLASP outperformed strong baselines on both datasets, demonstrating that synthetic data from a large LLM can effectively augment smaller models in low-resource semantic parsing.

Figura 2: Comparing “flat” semantics (Slot Tagging and Intent Classification, upper) to Semantic Parsing (lower).