[Paper Review] Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

Alexander Rietzler, Sebastian Stabinger+2 more|arXiv (Cornell University)|Aug 30, 2019

Sentiment Analysis and Opinion Mining20 references132 citations

TL;DR

The paper presents a two-step approach using domain-specific fine-tuning of BERT followed by supervised ATSC training, achieving new state-of-the-art results on SemEval 2014 restaurants dataset and demonstrating robust cross-domain transfer from laptops to restaurants and vice versa.

ABSTRACT

Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect-Based Sentiment Analysis (ABSA), which has many applications e.g. in e-commerce, where data and insights from reviews can be leveraged to create value for businesses and customers. Recently, deep transfer-learning methods have been applied successfully to a myriad of Natural Language Processing (NLP) tasks, including ATSC. Building on top of the prominent BERT language model, we approach ATSC using a two-step procedure: self-supervised domain-specific BERT language model finetuning, followed by supervised task-specific finetuning. Our findings on how to best exploit domain-specific language model finetuning enable us to produce new state-of-the-art performance on the SemEval 2014 Task 4 restaurants dataset. In addition, to explore the real-world robustness of our models, we perform cross-domain evaluation. We show that a cross-domain adapted BERT language model performs significantly better than strong baseline models like vanilla BERT-base and XLNet-base. Finally, we conduct a case study to interpret model prediction errors.

Motivation & Objective

• Motivate ATSC as a fine-grained sentiment task within ABSA and address domain shift in real-world applications.
• Investigate how domain-specific self-supervised fine-tuning of BERT impacts ATSC performance.
• Evaluate cross-domain transfer and joint-domain training to assess robustness across restaurant and laptop domains.

Proposed method

• Two-step procedure: domain-specific self-supervised finetuning of BERT language model, then supervised ATSC fine-tuning.
• Transform ATSC into BERT sequence-pair classification by feeding [CLS] sentence [SEP] target [SEP].
• Systematically vary finetuning domain (restaurants, laptops, or both) and training setup (in-domain, cross-domain, joint-domain).
• Use input reduction to interpret model predictions and analyze errors.

Experimental results

Research questions

• RQ1 RQ1: How does the amount of BERT language model finetuning on domain data affect ATSC performance?
• RQ2 RQ2: What ATSC performance is achievable with in-domain training when exploiting full domain-specific finetuning?
• RQ3 RQ3: How does cross-domain adaptation (domain-adapted LM finetuning) affect ATSC performance compared to in-domain baselines?
• RQ4 RQ4: Does joint-domain training improve performance across domains?
• RQ5 RQ5: What error types remain after domain adaptation, and how can interpretation help understand them?

Key findings

• Domain-specific LM finetuning significantly boosts ATSC accuracy, with diminishing gains after tens of millions of finetuned sentences depending on domain.
• In-domain results with domain-adapted LMs approach state-of-the-art on restaurants and laptops, notably achieving new SOTA on restaurants (87.14% accuracy).
• Cross-domain adaptation (target-domain LM finetuned models) yields consistent gains over BERT-base, e.g., +2.2–3.6 percentage points depending on source/target pairing.
• Joint-domain training (Laptops+Restaurants) further improves macro-F1, especially in imbalanced datasets where neutral class performance benefits.
• Case study via input reduction reveals that domain-adapted models better capture context-sensitive sentiment expressions and negations, but neutral cases remain challenging.