[Paper Review] Helping each Other: A Framework for Customer-to-Customer Suggestion Mining using a Semi-supervised Deep Neural Network

Hitesh Golchha, Deepak Gupta+2 more|arXiv (Cornell University)|Nov 1, 2018

Sentiment Analysis and Opinion Mining15 references4 citations

TL;DR

This paper proposes a semi-supervised hybrid deep learning model for customer-to-customer suggestion mining in reviews, combining CNN, LSTM, and self-training to improve detection of suggestive sentences. It achieves SOTA F-scores of 65.6% (hotel) and 65.5% (electronics) by leveraging unlabeled data, marking the first use of self-training for this task.

ABSTRACT

Suggestion mining is increasingly becoming an important task along with sentiment analysis. In today's cyberspace world, people not only express their sentiments and dispositions towards some entities or services, but they also spend considerable time sharing their experiences and advice to fellow customers and the product/service providers with two-fold agenda: helping fellow customers who are likely to share a similar experience, and motivating the producer to bring specific changes in their offerings which would be more appreciated by the customers. In our current work, we propose a hybrid deep learning model to identify whether a review text contains any suggestion. The model employs semi-supervised learning to leverage the useful information from the large amount of unlabeled data. We evaluate the performance of our proposed model on a benchmark customer review dataset, comprising of the reviews of Hotel and Electronics domains. Our proposed approach shows the F-scores of 65.6% and 65.5% for the Hotel and Electronics review datasets, respectively. These performances are significantly better compared to the existing state-of-the-art system.

Motivation & Objective

• To develop a robust method for identifying suggestive sentences in customer reviews, particularly those offering advice to fellow customers.
• To address the scarcity of labeled data in suggestion mining by leveraging large-scale unlabeled reviews through semi-supervised learning.
• To improve upon existing state-of-the-art models by integrating linguistic features and deep neural architectures.
• To explore the effectiveness of self-training in low-resource suggestion mining scenarios.

Proposed method

• A hybrid neural architecture combines CNN and LSTM to extract hierarchical and sequential features from review sentences.
• The model employs self-training (a form of semi-supervised learning) to iteratively label high-confidence predictions from unlabeled data and retrain the model.
• Linguistically motivated features such as POS tags, dependency relations, sentiment, and sequential patterns are integrated into the input representation.
• The framework is trained and evaluated on benchmark datasets from the hotel and electronics domains, with class imbalance addressed through careful sampling and analysis.
• A confusion matrix and qualitative error analysis are used to diagnose failure cases and guide model refinement.

Experimental results

Research questions

• RQ1 Can a hybrid deep learning model with semi-supervised learning outperform existing supervised baselines in customer-to-customer suggestion mining?
• RQ2 How effective is self-training in improving model generalization when labeled data is limited?
• RQ3 What types of linguistic patterns and syntactic structures are most predictive of suggestive intent in customer reviews?
• RQ4 Why do certain false positives and false negatives occur, and can they be mitigated with better data or architecture?

Key findings

• The proposed hybrid model with self-training achieves an F-score of 65.6% on the hotel review dataset and 65.5% on the electronics review dataset, surpassing the current state-of-the-art.
• Semi-supervised learning significantly improves performance by effectively utilizing large volumes of unlabeled customer reviews.
• False positives often arise from imperative-like constructions that are not actual suggestions, such as second-person narratives or non-imperative commands.
• False negatives are commonly caused by long, complex sentences where only a clause is suggestive, or when two sentences are merged without proper punctuation.
• The model struggles with sarcasm and subtle suggestions, especially when sarcasm is mislabeled in the training data.
• Error analysis reveals that systems learn to detect suggestive terms and imperative mood, but require more data to better distinguish between similar syntactic forms.