[Paper Review] Conditional Augmentation for Aspect Term Extraction via Masked Sequence-to-Sequence Generation

Kun Li, Chengbo Chen+3 more|arXiv (Cornell University)|Apr 30, 2020

Sentiment Analysis and Opinion Mining34 references4 citations

TL;DR

This paper proposes a controllable data augmentation method for aspect term extraction using masked sequence-to-sequence generation, where a model reconstructs masked segments of review sentences while preserving original aspect labels. The approach significantly improves model performance on low-resource ATE benchmarks by generating diverse, fluent, and label-consistent training examples.

ABSTRACT

Aspect term extraction aims to extract aspect terms from review texts as opinion targets for sentiment analysis. One of the big challenges with this task is the lack of sufficient annotated data. While data augmentation is potentially an effective technique to address the above issue, it is uncontrollable as it may change aspect words and aspect labels unexpectedly. In this paper, we formulate the data augmentation as a conditional generation task: generating a new sentence while preserving the original opinion targets and labels. We propose a masked sequence-to-sequence method for conditional augmentation of aspect term extraction. Unlike existing augmentation approaches, ours is controllable and allows us to generate more diversified sentences. Experimental results confirm that our method alleviates the data scarcity problem significantly. It also effectively boosts the performances of several current models for aspect term extraction.

Motivation & Objective

• Address the challenge of data scarcity in aspect term extraction (ATE) by developing a controllable data augmentation technique.
• Ensure generated sentences preserve original aspect term labels and maintain the original opinion targets as valid aspect terms.
• Overcome limitations of existing methods—such as GANs, VAEs, and word replacement—by enabling more diverse and contextually coherent sentence generation.
• Demonstrate that conditional generation via masked sequence-to-sequence modeling outperforms alternative language models like BERT and GPT-2 in this specific augmentation task.
• Establish a framework applicable beyond ATE to other low-resource sequence labeling tasks such as named entity recognition and chunking.

Proposed method

• Formulate data augmentation as a conditional sequence-to-sequence generation task, where the model generates a full sentence given a partially masked input and its corresponding label sequence.
• Use a Transformer-based encoder-decoder architecture with masked sequence-to-sequence pre-training, similar to MASS, to enable context-aware and label-conditioned generation.
• Mask multiple consecutive tokens in the input sentence and train the model to reconstruct the masked span using both the context and the label sequence as conditioning signals.
• Integrate label information into the encoder input to ensure that aspect terms remain valid opinion targets in the generated sentences.
• Train the model end-to-end using a standard cross-entropy loss on the reconstruction task, enabling auto-regressive generation of diverse and fluent sentences.
• Compare the proposed method with replacement-based augmentation (using WordNet synonyms) and fine-tuned BERT/GPT-2 variants to validate superiority in fluency, diversity, and label consistency.

Experimental results

Research questions

• RQ1 Can masked sequence-to-sequence generation be effectively used to generate diverse, fluent, and label-consistent sentences for aspect term extraction data augmentation?
• RQ2 How does the proposed method compare to GANs, VAEs, and word replacement-based approaches in preserving aspect term labels and maintaining sentence coherence?
• RQ3 Why is the proposed encoder-decoder framework with masked sequence-to-sequence learning superior to BERT and GPT-2 for this conditional generation task?
• RQ4 To what extent does the proposed augmentation improve the performance of downstream ATE models on low-resource datasets?
• RQ5 Can this framework be generalized to other low-resource sequence labeling tasks beyond aspect term extraction?

Key findings

• The proposed method significantly improves F1 scores on two benchmark ATE datasets by generating more diverse and label-consistent training examples.
• The model achieves higher fluency and BLEU scores than replacement-based methods, which are limited by synonym availability and context-agnostic word swaps.
• GPT-2 performs poorly due to lack of an encoder, resulting in uncontrollable generation that fails to align with label sequences.
• BERT performs worse than the proposed method due to its independent token reconstruction assumption, which leads to incoherent sequences.
• Qualitative analysis confirms that the method generates semantically varied and contextually appropriate sentences while preserving aspect terms as opinion targets.
• The framework is transferable and effective for other low-resource sequence labeling tasks, such as named entity recognition and chunking.