[Paper Review] Sentiment Analysis: Predicting Yelp Scores

Bhanu Prakash Reddy Guda, Mashrin Srivastava+1 more|arXiv (Cornell University)|Jan 20, 2022

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes a novel multi-tasked joint BERT model that fuses Yelp review text and meta-features to predict sentiment, achieving superior performance over baselines. The model leverages attention mechanisms and joint learning with sentiment scores, demonstrating improved accuracy and interpretability while maintaining balanced performance across restaurants with varying review counts.

ABSTRACT

In this work, we predict the sentiment of restaurant reviews based on a subset of the Yelp Open Dataset. We utilize the meta features and text available in the dataset and evaluate several machine learning and state-of-the-art deep learning approaches for the prediction task. Through several qualitative experiments, we show the success of the deep models with attention mechanism in learning a balanced model for reviews across different restaurants. Finally, we propose a novel Multi-tasked joint BERT model that improves the overall classification performance.

Motivation & Objective

• To develop a robust sentiment prediction model for Yelp restaurant reviews using both textual and meta-features.
• To evaluate the effectiveness of deep learning models, particularly BERT with attention, in capturing nuanced sentiment across diverse restaurant reviews.
• To investigate whether joint learning with sentiment scores improves classification performance and model generalization.
• To analyze model interpretability through attention visualization and identify failure cases related to implicit sentiment cues.
• To ensure balanced performance across restaurants with low or high review volumes, avoiding data bias.

Proposed method

• The model employs a multi-tasked joint BERT architecture that takes both review text and meta-features as input for sentiment classification.
• It uses attention mechanisms to highlight salient words in the text that contribute to sentiment prediction, improving interpretability.
• The model is jointly trained on two objectives: predicting star ratings (binary: 1–3 vs. 4–5) and predicting the AFINN-based sentiment score.
• Feature engineering includes dropping binary word occurrence features to prevent overfitting and selecting only relevant features correlated with sentiment.
• The dataset is split into 70% training, 15% validation, and 15% test sets, with pre-processing to remove irrelevant features like reviewer name and character count.
• Model training uses early stopping to prevent overfitting, and performance is evaluated via accuracy, loss curves, and confusion matrices.

Experimental results

Research questions

• RQ1 How does joint modeling of meta-features and text improve sentiment prediction performance compared to using text alone?
• RQ2 Why does the model perform worse in 5-star classification than in binary sentiment classification, and what causes this confusion?
• RQ3 Can attention mechanisms effectively highlight sentiment-relevant words, and does this improve model interpretability?
• RQ4 Does the model exhibit bias toward restaurants with more reviews, and how is performance balanced across low- and high-volume restaurants?
• RQ5 What types of failure cases emerge, particularly in reviews with implicit sentiment or external knowledge dependencies?

Key findings

• The multi-tasked joint BERT model outperforms all baselines in both binary and 5-star sentiment classification, demonstrating state-of-the-art performance on the Yelp dataset.
• Attention mechanisms successfully highlight sentiment-significant words such as 'perfect' and 'bad', confirming the model's interpretability and alignment with human judgment.
• The model shows no significant bias toward restaurants with high review counts, maintaining balanced performance across all restaurants.
• Confusion in 5-star classification is primarily between adjacent classes (e.g., 3 vs. 4 stars), indicating difficulty in distinguishing subtle sentiment differences.
• Failure cases occur in reviews relying on implicit sentiment cues (e.g., cooking metaphors), where external knowledge is required, suggesting a need for knowledge graph integration.
• Training loss decreases steadily, and validation loss stabilizes, confirming effective convergence with early stopping preventing overfitting.