[Paper Review] LINDA: Unsupervised Learning to Interpolate in Natural Language Processing

Yekyung Kim, Seohyeong Jeong+1 more|arXiv (Cornell University)|Dec 28, 2021

Natural Language Processing Techniques4 citations

TL;DR

LINDA proposes an unsupervised neural method for interpolating between natural language sentences without relying on heuristics or manual resources. By training a conditional language model to generate interpolated texts based on mixing ratios, LINDA enables effective mixup-based data augmentation, significantly improving generalization in text classification across in-domain, out-of-domain, and low-resource settings.

ABSTRACT

Despite the success of mixup in data augmentation, its applicability to natural language processing (NLP) tasks has been limited due to the discrete and variable-length nature of natural languages. Recent studies have thus relied on domain-specific heuristics and manually crafted resources, such as dictionaries, in order to apply mixup in NLP. In this paper, we instead propose an unsupervised learning approach to text interpolation for the purpose of data augmentation, to which we refer as "Learning to INterpolate for Data Augmentation" (LINDA), that does not require any heuristics nor manually crafted resources but learns to interpolate between any pair of natural language sentences over a natural language manifold. After empirically demonstrating the LINDA's interpolation capability, we show that LINDA indeed allows us to seamlessly apply mixup in NLP and leads to better generalization in text classification both in-domain and out-of-domain.

Motivation & Objective

• To address the challenge of applying mixup in NLP due to the discrete and variable-length nature of text.
• To eliminate reliance on domain-specific heuristics or manually curated resources like dictionaries for text interpolation.
• To learn a neural interpolation operator that produces semantically coherent, interpolated sentences over the natural language manifold.
• To evaluate whether such learned interpolation enables effective data augmentation via mixup in text classification tasks.

Proposed method

• Define text interpolation as sampling from a conditional language model conditioned on two input sentences and a mixing ratio α ∈ [0,1].
• Train a sequence-to-sequence model (fine-tuned BART) to generate interpolated text where both input sentences are highly probable, with relative likelihood modulated by α.
• Use a contrastive learning objective to ensure interpolated outputs are close to both inputs in latent space while maintaining diversity.
• Apply the learned interpolation as a drop-in replacement in mixup training, generating mixed inputs and soft labels via f_int^x and f_int^y.
• Optimize the model using a combination of cross-entropy loss and a contrastive loss to preserve semantic fidelity to both source sentences.
• Fine-tune the model on a moderate Wikipedia corpus to learn interpolation patterns across diverse sentence pairs.

Experimental results

Research questions

• RQ1 Can a neural network learn to interpolate between arbitrary pairs of natural language sentences in a way that is both semantically coherent and controllable by a mixing ratio?
• RQ2 Does unsupervised interpolation via LINDA enable effective data augmentation in NLP when plugged into mixup?
• RQ3 How does LINDA compare to heuristic and model-based data augmentation methods in terms of in-domain and out-of-domain generalization?
• RQ4 Can LINDA improve model performance in low-resource settings where labeled data is scarce?
• RQ5 Is the interpolation behavior monotonic and predictable with respect to the mixing ratio α?

Key findings

• LINDA generates semantically coherent interpolated sentences across diverse input pairs, with qualitative inspection confirming controlled interpolation based on mixing ratio.
• Automated metrics show a monotonic relationship between the mixing ratio α and similarity to the source sentences, confirming controllability.
• In low-resource settings (5 or 10 examples per class), LINDA outperforms EDA, SSMBA, and mixup on all datasets except TREC-fine (5-shot), where performance is within standard deviation.
• In full-resource settings, LINDA achieves the highest accuracy on four out of seven datasets and is competitive on the rest, outperforming non-augmented baselines consistently.
• On NLI tasks (RTE, QNLI, MNLI), LINDA improves accuracy by 5.4% on RTE, 0.7% on QNLI, and 0.5% on MNLI, while other methods degrade performance.
• In out-of-domain generalization, LINDA improves performance across all domains (Amazon, Yelp, Movies) and consistently exceeds the vanilla baseline, unlike EDA and mixup which show no improvement.