[Paper Review] Exploiting BERT for End-to-End Aspect-based Sentiment Analysis

Xin Li, Lidong Bing+2 more|arXiv (Cornell University)|Oct 2, 2019

Topic Modeling82 references47 citations

TL;DR

The paper shows that BERT with simple downstream layers can outperform prior E2E-ABSA models, establishing a BERT-based benchmark and highlighting fine-tuning importance. It evaluates several BERT-based architectures and demonstrates robustness to overfitting on two SemEval-derived datasets.

ABSTRACT

In this paper, we investigate the modeling power of contextualized embeddings from pre-trained language models, e.g. BERT, on the E2E-ABSA task. Specifically, we build a series of simple yet insightful neural baselines to deal with E2E-ABSA. The experimental results show that even with a simple linear classification layer, our BERT-based architecture can outperform state-of-the-art works. Besides, we also standardize the comparative study by consistently utilizing a hold-out validation dataset for model selection, which is largely ignored by previous works. Therefore, our work can serve as a BERT-based benchmark for E2E-ABSA.

Motivation & Objective

• Assess the effectiveness of contextualized embeddings from BERT for End-to-End Aspect-based Sentiment Analysis (E2E-ABSA).
• Investigate simple downstream architectures atop BERT (linear, RNN, self-attention, CRF) for sequence labeling in E2E-ABSA.
• Provide a standardized evaluation setup with a hold-out development set for model selection.
• Offer a BERT-based benchmark for E2E-ABSA and release open-source code for reproducibility.

Proposed method

• Embed tokens with a pre-trained BERT model to obtain contextualized representations H^L for input tokens.
• Attach a simple E2E-ABSA layer on top of BERT, exploring linear, GRU, self-attention (SAN), and transformer-based (TFM) variants.
• Optionally add a linear-chain CRF layer to model sequence dependencies via Viterbi decoding.
• Fine-tune BERT (vs. using it as a fixed feature extractor) to maximize task performance.
• Compare with traditional LSTM-CRF and other ABSA baselines on LAPTOP and REST datasets.
• Report results using micro-averaged F1 scores on development and test sets.

Experimental results

Research questions

• RQ1 Can BERT contextualized embeddings alone (with a simple linear classifier) outperform prior E2E-ABSA models?
• RQ2 Do more powerful downstream layers (GRU, SAN, transformer-like, CRF) yield significant gains over a linear classifier when using BERT representations?
• RQ3 Is fine-tuning BERT essential for optimal E2E-ABSA performance, or can BERT be used as a fixed feature extractor?
• RQ4 How robust are BERT-based E2E-ABSA models to overfitting on small datasets?
• RQ5 How do BERT-based approaches compare to existing state-of-the-art models on standard ABSA benchmarks?

Key findings

• A simple BERT+Linear model already outperforms many existing ABSA methods without BERT.
• More advanced downstream layers (GRU, SAN, TFM, CRF) provide additional gains in F1 score.
• BERT-based models show robustness to overfitting, with stable F1 during extended training.
• Fine-tuning BERT yields substantially better results than using BERT as a fixed feature extractor.
• On LAPTOP and REST datasets, BERT-based models with SAN/TFM achieve the highest reported F1 scores in the study.