[Paper Review] Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data

Alana Marzoev, Samuel Madden+3 more|arXiv (Cornell University)|Apr 28, 2020

Topic Modeling28 references19 citations

TL;DR

This paper proposes a simulation-to-real transfer framework for natural language processing that trains models on synthetic language data generated via rule-based grammars and generalizes to natural language by projecting real utterances onto the synthetic language space using pretrained sentence embeddings. The approach achieves state-of-the-art performance on semantic parsing and instruction-following tasks without any natural training data, outperforming models fine-tuned on human-annotated datasets in some cases.

ABSTRACT

Large, human-annotated datasets are central to the development of natural language processing models. Collecting these datasets can be the most challenging part of the development process. We address this problem by introducing a general purpose technique for ``simulation-to-real'' transfer in language understanding problems with a delimited set of target behaviors, making it possible to develop models that can interpret natural utterances without natural training data. We begin with a synthetic data generation procedure, and train a model that can accurately interpret utterances produced by the data generator. To generalize to natural utterances, we automatically find projections of natural language utterances onto the support of the synthetic language, using learned sentence embeddings to define a distance metric. With only synthetic training data, our approach matches or outperforms state-of-the-art models trained on natural language data in several domains. These results suggest that simulation-to-real transfer is a practical framework for developing NLP applications, and that improved models for transfer might provide wide-ranging improvements in downstream tasks.

Motivation & Objective

• To address the high cost and difficulty of collecting large, human-annotated natural language datasets for NLP applications.
• To enable effective generalization from synthetic language data to real-world natural language utterances in language understanding tasks.
• To develop a practical, scalable alternative to end-to-end training that avoids the need for extensive human-annotation while maintaining high performance.
• To demonstrate that projection-based transfer using pretrained sentence embeddings can outperform direct transfer of representations in synthetic-to-real NLP transfer.
• To provide a reusable framework for building grounded language understanding systems in new domains with minimal annotation effort.

Proposed method

• Synthetic training data is generated using high-precision, rule-based grammars that map target behaviors (e.g., actions, queries) to natural-sounding utterances.
• A model is trained on this synthetic data to accurately interpret utterances within the defined language distribution.
• Pretrained sentence embeddings (e.g., BERT) are used to learn a meaning-preserving distance metric between natural and synthetic sentences.
• At inference time, each natural utterance is projected to the nearest synthetic utterance in the embedding space, preserving semantic meaning.
• The projected synthetic utterance is then interpreted by the trained model, enabling generalization to real-world language.
• The framework is evaluated on semantic parsing and instruction-following benchmarks, with ablation studies to validate the effectiveness of the projection mechanism.

Experimental results

Research questions

• RQ1 Can a model trained solely on synthetic language data achieve performance comparable to models trained on large-scale natural language datasets?
• RQ2 How effective is sentence embedding-based projection in transferring knowledge from synthetic to natural language distributions?
• RQ3 Does using pretrained representations to find paraphrases in the synthetic space outperform direct transfer of representations in end-to-end models?
• RQ4 Can grammar-engineered synthetic data combined with embedding-based projection reduce the need for human-annotated data in grounded language understanding?
• RQ5 What is the impact of data distribution shifts on model generalization in simulation-to-real NLP transfer?

Key findings

• On eight semantic parsing datasets, the sim-to-real approach matched or exceeded the performance of a supervised semantic parser on three of the eight tasks using no natural training data.
• On a grounded instruction-following benchmark in a gridworld environment, the method surpassed the performance of a standard model fine-tuned with human annotations.
• The use of sentence embeddings for projecting natural utterances onto the synthetic language space enabled high-accuracy interpretation despite distributional shift.
• Direct transfer of pretrained representations from synthetic to natural data was less effective than using the embeddings to find paraphrases in the synthetic space.
• The framework enables developers to build high-accuracy language understanding systems in new domains using only synthetic data and pretraining, significantly reducing annotation burden.
• The approach demonstrates that sim-to-real transfer is a viable and effective paradigm for NLP, especially in data-scarce or interactive application settings.