[Paper Review] BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis

Hu Xu, Bing Liu+2 more|arXiv (Cornell University)|Apr 3, 2019

Sentiment Analysis and Opinion Mining44 references358 citations

TL;DR

The paper introduces ReviewRC for review-based reading comprehension and presents a joint post-training approach (BERT-PT) that adapts BERT with domain and task knowledge to improve RRC, AE, and ASC. It shows that vanilla BERT underperforms on reviews, while BERT-PT achieves state-of-the-art results across tasks and domains.

ABSTRACT

Question-answering plays an important role in e-commerce as it allows potential customers to actively seek crucial information about products or services to help their purchase decision making. Inspired by the recent success of machine reading comprehension (MRC) on formal documents, this paper explores the potential of turning customer reviews into a large source of knowledge that can be exploited to answer user questions.~We call this problem Review Reading Comprehension (RRC). To the best of our knowledge, no existing work has been done on RRC. In this work, we first build an RRC dataset called ReviewRC based on a popular benchmark for aspect-based sentiment analysis. Since ReviewRC has limited training examples for RRC (and also for aspect-based sentiment analysis), we then explore a novel post-training approach on the popular language model BERT to enhance the performance of fine-tuning of BERT for RRC. To show the generality of the approach, the proposed post-training is also applied to some other review-based tasks such as aspect extraction and aspect sentiment classification in aspect-based sentiment analysis. Experimental results demonstrate that the proposed post-training is highly effective. The datasets and code are available at https://www.cs.uic.edu/~hxu/.

Motivation & Objective

• Define Review Reading Comprehension (RRC) as extracting answer spans from product reviews for user questions.
• Create ReviewRC, a domain-aligned extractive RRC dataset based on SemEval reviews.
• Develop a joint post-training strategy (domain knowledge + MRC task knowledge) to adapt BERT to reviews.
• Demonstrate that post-training improves RRC, AE, and ASC beyond vanilla BERT on laptop and restaurant domains.

Proposed method

• Formulate RRC as a span prediction task with inputs [CLS], question, [SEP], review, [SEP], producing start and end pointers to select the answer span.
• Introduce ReviewRC dataset annotation following SQuAD style for laptop and restaurant reviews.
• Propose BERT post-training (BERT-PT) combining domain knowledge (via MLM and NSP on reviews) and task knowledge (via MRC data from SQuAD) with gradient accumulation to manage memory.
• Use Algorithm 1 to perform joint post-training by alternating domain knowledge and MRC batches and accumulate gradients.
• Apply BERT fine-tuning on three end tasks: RRC, Aspect Extraction (AE), and Aspect Sentiment Classification (ASC).
• Compare against baselines including DrQA, DrQA+MRC, and various BERT variants (BERT, BERT-DK, BERT-MRC, BERT-PT).

Experimental results

Research questions

• RQ1 RQ1: How much does post-training improve RRC, AE, and ASC compared with state-of-the-art baselines?
• RQ2 RQ2: How does vanilla BERT perform on review-based tasks without domain/task adaptation?
• RQ3 RQ3: What are the contributions of domain knowledge vs. task-aware knowledge to gains from post-training?

Key findings

• BERT-PT yields the best performance across all three tasks and domains (Laptop and Restaurant).
• Vanilla BERT performs worse on review-based tasks, highlighting the need for adaptation.
• Domain knowledge post-training primarily boosts AE, while task-aware post-training (MRC) provides large gains for RRC and ASC depending on domain.
• MRC data aids ASC by framing polarity as a special MRC problem; boundary accuracy improves notably for RRC.
• Domain knowledge benefits Restaurant more than Laptop, suggesting domain coverage differences between Wikipedia and review domains.
• Ablations show distinct contributions from domain vs. task knowledge across tasks and domains.