[Paper Review] Can ChatGPT's Responses Boost Traditional Natural Language Processing?

Mostafa M. Amin, Erik Cambria+1 more|arXiv (Cornell University)|Jul 6, 2023

ECG Monitoring and AnalysisMedicine3 citations

TL;DR

This paper investigates whether verbose responses from ChatGPT contain novel knowledge that can enhance traditional NLP models in affective computing tasks. By prompting ChatGPT for detailed reasoning on sentiment analysis, suicide detection, and personality assessment, and fusing these responses with RoBERTa or BoW features via early or late fusion, the study shows that ChatGPT's verbose outputs provide unique information that boosts performance beyond standalone models, especially when combined with fine-tuned NLP techniques.

ABSTRACT

The employment of foundation models is steadily expanding, especially with the launch of ChatGPT and the release of other foundation models. These models have shown the potential of emerging capabilities to solve problems, without being particularly trained to solve. A previous work demonstrated these emerging capabilities in affective computing tasks; the performance quality was similar to traditional Natural Language Processing (NLP) techniques, but falling short of specialised trained models, like fine-tuning of the RoBERTa language model. In this work, we extend this by exploring if ChatGPT has novel knowledge that would enhance existing specialised models when they are fused together. We achieve this by investigating the utility of verbose responses from ChatGPT about solving a downstream task, in addition to studying the utility of fusing that with existing NLP methods. The study is conducted on three affective computing problems, namely sentiment analysis, suicide tendency detection, and big-five personality assessment. The results conclude that ChatGPT has indeed novel knowledge that can improve existing NLP techniques by way of fusion, be it early or late fusion.

Motivation & Objective

• To investigate whether ChatGPT's verbose responses contain novel knowledge not present in fine-tuned NLP models like RoBERTa.
• To evaluate the effectiveness of fusing ChatGPT's verbose outputs with traditional NLP features (RoBERTa and BoW) for affective computing tasks.
• To compare early fusion (feature-level concatenation) and late fusion (prediction averaging) in improving model performance.
• To address the challenge of parsing non-uniform label formatting from non-verbose ChatGPT responses by leveraging verbose reasoning.
• To demonstrate that processing verbose responses with NLP techniques enables reliable fusion and improved performance over label-parsing baselines.

Proposed method

• Prompting ChatGPT to generate verbose, step-by-step reasoning responses for three binary classification tasks: sentiment analysis, suicide tendency detection, and big-five personality assessment.
• Extracting contextual features from both the original input text and the ChatGPT-generated verbose response using RoBERTa-base and n-gram Bag-of-Words (BoW) representations.
• Training separate Multi-Layer Perceptrons (MLPs) on the extracted features for each modality (e.g., Text+RoBERTa, ChatGPT+RoBERTa).
• Implementing early fusion by concatenating feature vectors from multiple modalities before final prediction.
• Implementing late fusion by averaging the prediction probabilities from individual models across modalities.
• Using a baseline that parses only the final label from non-verbose ChatGPT responses to compare performance against verbose response processing.

Experimental results

Research questions

• RQ1 Can ChatGPT’s verbose responses provide novel knowledge that improves performance on affective computing tasks beyond what specialized NLP models offer?
• RQ2 Does fusing features from ChatGPT’s verbose responses with traditional NLP features (RoBERTa or BoW) lead to better performance than using either modality alone?
• RQ3 Is early fusion (feature-level concatenation) or late fusion (prediction-level averaging) more effective for combining ChatGPT and NLP model outputs?
• RQ4 How does processing verbose responses with NLP techniques compare to directly parsing labels from non-verbose responses in terms of performance and reliability?
• RQ5 In which tasks does the fusion of ChatGPT and NLP models yield the greatest improvement, and what factors influence this?

Key findings

• ChatGPT’s verbose responses contain novel, useful knowledge that enhances traditional NLP models in affective computing tasks, particularly when fused with RoBERTa or BoW features.
• Late fusion consistently outperformed early fusion in sentiment analysis, while early fusion was superior in suicide detection, indicating task-dependent optimal fusion strategies.
• The RoBERTa-based model on the original text alone achieved the best performance in big-five personality assessment, suggesting that ChatGPT’s responses were less reliable for this task.
• Processing verbose responses with NLP techniques significantly improved performance over the baseline of directly parsing labels from non-verbose outputs, which suffered from inconsistent formatting.
• The fusion of ChatGPT’s verbose responses with RoBERTa features—especially via early fusion—yielded the highest overall performance across tasks, demonstrating the value of combining large language model reasoning with fine-tuned sequence encoders.
• Using RoBERTa to process both the original text and the ChatGPT response was sufficient to achieve state-of-the-art performance in most cases, reducing the need for complex fusion architectures.