[Paper Review] Auto Analysis of Customer Feedback using CNN and GRU Network

Deepak Gupta, Pabitra Lenka+3 more|arXiv (Cornell University)|Oct 12, 2017

Sentiment Analysis and Opinion Mining12 references3 citations

TL;DR

This paper proposes a deep learning approach combining Convolutional Neural Networks (CNN) and Gated Recurrent Units (GRU) for automatic customer feedback classification across four languages—English, French, Spanish, and Japanese—without manual feature engineering. The model achieves second place in French (71.75% accuracy) and top three in other languages on the IJCNLP shared task, demonstrating strong multilingual performance despite class imbalance and short feedback challenges.

ABSTRACT

Analyzing customer feedback is the best way to channelize the data into new marketing strategies that benefit entrepreneurs as well as customers. Therefore an automated system which can analyze the customer behavior is in great demand. Users may write feedbacks in any language, and hence mining appropriate information often becomes intractable. Especially in a traditional feature-based supervised model, it is difficult to build a generic system as one has to understand the concerned language for finding the relevant features. In order to overcome this, we propose deep Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) based approaches that do not require handcrafting of features. We evaluate these techniques for analyzing customer feedback sentences in four languages, namely English, French, Japanese and Spanish. Our empirical analysis shows that our models perform well in all the four languages on the setups of IJCNLP Shared Task on Customer Feedback Analysis. Our model achieved the second rank in French, with an accuracy of 71.75% and third ranks for all the other languages.

Motivation & Objective

• To develop a generic, automated system for classifying customer feedback into six predefined categories: comment, request, bug, complaint, meaningless, and undetermined.
• To overcome the limitations of traditional supervised models that require language-specific feature engineering for each language and domain.
• To evaluate the performance of deep learning models—specifically CNN and CNN+GRU—on multilingual customer feedback data without relying on handcrafted features.
• To investigate the impact of language-specific pre-trained embeddings and data translation on model generalization across languages.
• To analyze failure cases and identify key challenges such as ambiguous, short, or incomplete feedback sentences.

Proposed method

• The authors use a CNN-based architecture to extract local n-gram features from sentence embeddings, followed by a max-pooling layer to reduce dimensionality.
• A second model combines CNN with Gated Recurrent Units (GRU) to capture sequential dependencies in feedback sentences, improving context modeling.
• Word embeddings are initialized using pre-trained Google Word2Vec for English, while other languages use random initialization due to lack of available pre-trained embeddings.
• The system is trained on English feedback data and evaluated on both original-language test sets and English-translated versions of non-English test sets.
• The models are fine-tuned using cross-entropy loss and softmax classification for the six feedback categories.
• Error analysis is conducted on misclassified instances, categorized into ambiguous feedback, missing target entities, and overly short sentences.

Experimental results

Research questions

• RQ1 Can a deep learning model based on CNN and GRU effectively classify customer feedback across multiple languages without language-specific feature engineering?
• RQ2 How does performance vary when using translated test data versus original-language test data for low-resource languages like Japanese and Spanish?
• RQ3 What are the main failure modes of the model when classifying short, ambiguous, or incomplete feedback sentences?
• RQ4 To what extent does class imbalance in training data affect the model’s ability to detect minority classes such as 'bug'?
• RQ5 Can performance be further improved by using language-specific pre-trained word embeddings?

Key findings

• The CNN+GRU model achieved the second-highest accuracy in French (71.75%) and ranked third in English, Spanish, and Japanese on the IJCNLP shared task.
• The CNN model achieved 85.62% accuracy in Spanish and 63.00% in Japanese when using English-translated test data, showing moderate cross-lingual transferability.
• The model struggled most with the 'bug' class due to class imbalance, where complaints were far more frequent than bugs in training data.
• Ambiguous feedback, missing target entities, and very short sentences were the primary causes of misclassification, as identified in error analysis.
• Performance was limited by the lack of pre-trained word embeddings for non-English languages, suggesting that future improvements are possible with language-specific embeddings.
• Baseline systems using 3-gram features with SVM achieved significantly lower accuracy (e.g., 48.8% in English), confirming the advantage of deep learning approaches.