[Paper Review] How would Stance Detection Techniques Evolve after the Launch of ChatGPT?

Bowen Zhang, Daijun Ding+3 more|arXiv (Cornell University)|Dec 30, 2022

Topic Modeling68 citations

TL;DR

The paper investigates ChatGPT's zero-shot and in-domain performance on stance detection benchmarks (SemEval-2016 and P-Stance), showing ChatGPT can reach SOTA or near-SOTA results and can provide explanations for its predictions, potentially transforming the stance detection research paradigm.

ABSTRACT

Stance detection refers to the task of extracting the standpoint (Favor, Against or Neither) towards a target in given texts. Such research gains increasing attention with the proliferation of social media contents. The conventional framework of handling stance detection is converting it into text classification tasks. Deep learning models have already replaced rule-based models and traditional machine learning models in solving such problems. Current deep neural networks are facing two main challenges which are insufficient labeled data and information in social media posts and the unexplainable nature of deep learning models. A new pre-trained language model chatGPT was launched on Nov 30, 2022. For the stance detection tasks, our experiments show that ChatGPT can achieve SOTA or similar performance for commonly used datasets including SemEval-2016 and P-Stance. At the same time, ChatGPT can provide explanation for its own prediction, which is beyond the capability of any existing model. The explanations for the cases it cannot provide classification results are especially useful. ChatGPT has the potential to be the best AI model for stance detection tasks in NLP, or at least change the research paradigm of this field. ChatGPT also opens up the possibility of building explanatory AI for stance detection.

Motivation & Objective

• Assess ChatGPT's zero-shot stance detection performance on standard datasets (SemEval-2016, P-Stance).
• Evaluate ChatGPT's in-domain performance compared to traditional models trained on data.
• Analyze ChatGPT's ability to explain its stance decisions and its potential to enable explanatory AI for stance detection.

Proposed method

• Construct a direct prompt asking ChatGPT for the stance of a given tweet toward a target (zero-shot prompting).
• Evaluate outputs using F1-avg and macro-F1 (F1-m) on SemEval-2016 and P-Stance datasets.
• Compare zero-shot ChatGPT results with traditional models and with in-domain (80% training data) baselines.
• In cases where ChatGPT cannot provide a stance, analyze its explanations (for future explainable-AI insights).
• Update results using GPT-3.5-0301 variant in an updated version of the paper.

Experimental results

Research questions

• RQ1 Can ChatGPT achieve state-of-the-art or competitive stance detection performance in zero-shot settings on standard datasets?
• RQ2 How does ChatGPT perform in-domain compared to models trained with substantial labeled data?
• RQ3 What kinds of explanations does ChatGPT provide for its stance decisions, and can multi-turn prompting further improve performance?
• RQ4 What future research directions emerge from using ChatGPT for stance detection (prompt design, explainability, multi-round conversation)?

Key findings

• ChatGPT achieves state-of-the-art or similar performance to leading models on SemEval-2016 and P-Stance under zero-shot prompting.
• ChatGPT often outperforms baselines in zero-shot setups and remains competitive in in-domain setups.
• ChatGPT can provide explanations for its decisions, including cases with explicit and implicit stance signals.
• Chain-of-prompts (multi-turn interactions) can potentially further improve stance detection performance.
• Three suggested future directions include better prompt templates, leveraging ChatGPT's explanations for explanatory AI, and exploring multi-round conversations to improve coverage on difficult instances.