[Paper Review] Customer Sentiment Analysis using Weak Supervision for Customer-Agent Chat

Navdeep Jain|arXiv (Cornell University)|Nov 28, 2021

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes a weak supervision approach to train a RoBERTa-based sentiment classifier for customer-agent chat, using a combination of pre-trained sentiment models and domain-specific lexicon rules as labeling functions. The method achieves superior performance on domain-specific sentiment cases compared to off-the-shelf APIs like Google Cloud NLP, particularly in nuanced customer service contexts where context and terminology matter.

ABSTRACT

Prior work on sentiment analysis using weak supervision primarily focuses on different reviews such as movies (IMDB), restaurants (Yelp), products (Amazon).~One under-explored field in this regard is customer chat data for a customer-agent chat in customer support due to the lack of availability of free public data. Here, we perform sentiment analysis on customer chat using weak supervision on our in-house dataset. We fine-tune the pre-trained language model (LM) RoBERTa as a sentiment classifier using weak supervision. Our contribution is as follows:1) We show that by using weak sentiment classifiers along with domain-specific lexicon-based rules as Labeling Functions (LF), we can train a fairly accurate customer chat sentiment classifier using weak supervision. 2) We compare the performance of our custom-trained model with off-the-shelf google cloud NLP API for sentiment analysis. We show that by injecting domain-specific knowledge using LFs, even with weak supervision, we can train a model to handle some domain-specific use cases better than off-the-shelf google cloud NLP API. 3) We also present an analysis of how customer sentiment in a chat relates to problem resolution.

Motivation & Objective

• To develop a sentiment analysis model for customer-agent chat without relying on large-scale labeled data.
• To evaluate the effectiveness of weak supervision using pre-trained sentiment models and custom lexicon rules for domain-specific sentiment classification.
• To compare the performance of the weakly supervised model against off-the-shelf NLP APIs like Google Cloud NLP in real customer service scenarios.
• To analyze the relationship between sentiment evolution during chats and problem resolution outcomes.

Proposed method

• Employ five labeling functions: three general-purpose sentiment models (Textblob, AFINN, VADER) and two domain-specific lexicon-based rules for positive and negative sentiment.
• Use the Snorkel label model to aggregate weak labels into a single, more reliable label per instance, accounting for labeling function accuracy and correlation.
• Fine-tune RoBERTa using contrastive-regularized self-training (COSINE) on the weakly labeled data to improve model generalization.
• Train the model end-to-end using the Snorkel pipeline, which enables scalable weak supervision and label aggregation.
• Compare predictions from the fine-tuned RoBERTa model with those from Google Cloud NLP API on a held-out test set.

Experimental results

Research questions

• RQ1 Can weak supervision with domain-specific labeling functions improve sentiment classification accuracy in customer-agent chat compared to off-the-shelf APIs?
• RQ2 How does the integration of domain-specific lexicon rules affect model performance on nuanced sentiment expressions?
• RQ3 What is the relationship between sentiment trends during a chat and the resolution status of the customer’s issue?
• RQ4 To what extent do weak labels derived from multiple sources (e.g., Textblob, AFINN, and custom rules) improve final model performance?

Key findings

• The Snorkel label model using five labeling functions achieved a macro F1-score of 0.65 after fine-tuning RoBERTa, outperforming models trained with only Textblob (0.53) or AFINN (0.58).
• The RoBERTa model fine-tuned with weak supervision correctly classified 100% of the domain-specific sentiment cases where Google Cloud NLP API failed, such as competitor references and polite farewells.
• Chats ending with positive sentiment were strongly associated with issue resolution, while non-positive sentiment at the end indicated unresolved issues.
• When agents offered alternatives like call transfers or technician appointments, sentiment at the end varied (positive, neutral, or negative), indicating that sentiment is not always a direct proxy for resolution.
• The model trained with weak supervision showed higher accuracy than Google Cloud NLP API on domain-specific sentiment expressions, such as 'ATT has Fiber and it’s only $80' being interpreted as negative in a Comcast context.