[Paper Review] Predicting Helpfulness of Online Reviews

Abdalraheem Alsmadi, Shadi AlZu’bi+2 more|arXiv (Cornell University)|Aug 23, 2020

Sentiment Analysis and Opinion Mining21 references4 citations

TL;DR

This paper proposes a multi-approach machine learning framework to predict the helpfulness of online product reviews using supervised, semi-supervised, and pre-trained deep learning models on Amazon review data. It demonstrates that semi-supervised RCNN with FastText embeddings and fine-tuned BERT/RoBERTa models significantly outperform traditional classifiers, with the semi-supervised RCNN achieving 88.75% overall accuracy—6% higher than supervised RCNN—while BERT reaches 86.16% accuracy.

ABSTRACT

E-commerce dominates a large part of the world's economy with many websites dedicated to online selling products. The vast majority of e-commerce websites provide their customers with the ability to express their opinions about the products/services they purchase. These feedback in the form of reviews represent a rich source of information about the users' experiences and level of satisfaction, which is of great benefit to both the producer and the consumer. However, not all of these reviews are helpful/useful. The traditional way of determining the helpfulness of a review is through the feedback from human users. However, such a method does not necessarily cover all reviews. Moreover, it has many issues like bias, high cost, etc. Thus, there is a need to automate this process. This paper presents a set of machine learning (ML) models to predict the helpfulness online reviews. Mainly, three approaches are used: a supervised learning approach (using ML as well as deep learning (DL) models), a semi-supervised approach (that combines DL models with word embeddings), and pre-trained word embedding models that uses transfer learning (TL). The latter two approaches are among the unique aspects of this paper as they follow the recent trend of utilizing unlabeled text. The results show that the proposed DL approaches have superiority over the traditional existing ones. Moreover, the semi-supervised has a remarkable performance compared with the other ones.

Motivation & Objective

• To automate the prediction of helpfulness in online product reviews, reducing reliance on costly and biased human voting.
• To address the underutilization of unlabeled review data by integrating semi-supervised learning techniques.
• To evaluate the effectiveness of pre-trained language models like BERT and RoBERTa in improving classification accuracy.
• To compare traditional machine learning models, deep learning architectures, and transfer learning approaches on a standardized Amazon review dataset.
• To provide a scalable, accurate solution for e-commerce platforms to surface only the most useful reviews to consumers.

Proposed method

• Trained a supervised learning pipeline using multiple classifiers: FastText, SVM, Bi-LSTM, CNN, and RCNN on labeled Amazon review data.
• Implemented a semi-supervised RCNN model that leverages both labeled data (90k) and large-scale unlabeled data (490k) to improve generalization.
• Utilized FastText word embeddings with skip-gram architecture to enhance feature representation in the semi-supervised setting.
• Fine-tuned pre-trained transformer models BERT and RoBERTa using transfer learning on the same Amazon review dataset for end-to-end classification.
• Evaluated all models using accuracy as the primary metric across four product categories: Books, Electronics, CDs & Vinyl, and Movies & TV.
• Applied data augmentation and dynamic masking in BERT training to improve contextual representation and robustness.

Experimental results

Research questions

• RQ1 Can deep learning models like RCNN outperform traditional machine learning classifiers in predicting review helpfulness?
• RQ2 Does incorporating large-scale unlabeled data via semi-supervised learning improve the performance of review helpfulness prediction models?
• RQ3 To what extent do pre-trained language models like BERT and RoBERTa enhance prediction accuracy compared to fine-tuned RCNN and other classifiers?
• RQ4 How does the performance of semi-supervised RCNN compare to its supervised counterpart across different product categories?
• RQ5 Can transfer learning with BERT and RoBERTa achieve state-of-the-art results on the Amazon review helpfulness prediction task?

Key findings

• The supervised RCNN model achieved the highest accuracy among traditional classifiers, reaching 83.25% overall accuracy, outperforming FastText (75.9%), SVM (75.5%), Bi-LSTM (79.12%), and CNN (82%) on the same dataset.
• The semi-supervised RCNN model improved performance by 5–6 percentage points over the supervised RCNN across all categories, achieving 88.75% overall accuracy.
• The semi-supervised RCNN with FastText embeddings achieved 87% accuracy on Books, 86% on Electronics, 92% on CDs & Vinyl, and 90% on Movies & TV, significantly outperforming the supervised version.
• Pre-trained models BERT and RoBERTa achieved 86.16% and 84.12% overall accuracy, respectively, surpassing all supervised models including RCNN (83.25%) and outperforming even the semi-supervised RCNN in some categories.
• The BERT model achieved 86.3% accuracy on Books, 85.16% on Electronics, 87.7% on CDs & Vinyl, and 85.5% on Movies & TV, demonstrating strong generalization across diverse product types.
• The results confirm that unlabeled data and transfer learning significantly enhance model performance, with the semi-supervised RCNN and BERT models setting new benchmarks on this dataset.