[Paper Review] Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media

Bowen Zhang, Xianghua Fu+6 more|arXiv (Cornell University)|Apr 6, 2023

Misinformation and Its Impacts17 citations

TL;DR

The paper investigates using Chain-of-Thought prompting with ChatGPT (GPT-3.5) for stance detection on social media and finds CoT can achieve state-of-the-art or competitive results without training, while discussing biases and prompting design issues.

ABSTRACT

Stance detection predicts attitudes towards targets in texts and has gained attention with the rise of social media. Traditional approaches include conventional machine learning, early deep neural networks, and pre-trained fine-tuning models. However, with the evolution of very large pre-trained language models (VLPLMs) like ChatGPT (GPT-3.5), traditional methods face deployment challenges. The parameter-free Chain-of-Thought (CoT) approach, not requiring backpropagation training, has emerged as a promising alternative. This paper examines CoT's effectiveness in stance detection tasks, demonstrating its superior accuracy and discussing associated challenges.

Motivation & Objective

• Motivate the study by addressing deployment challenges of VLPLMs for stance detection on social media.
• Evaluate parameter-free Chain-of-Thought prompting (DQA and StSQA) with ChatGPT for stance prediction without training.
• Compare CoT-based prompting against traditional methods and other baselines on multiple datasets.
• Identify limitations and bias issues that affect CoT performance in stance detection.

Proposed method

• Compare two CoT prompting strategies: direct question-answering (DQA) and step-by-step question-answering (StSQA).
• Use zero-shot setups (DQA) and one-shot prompts (StSQA) to elicit stance judgments from ChatGPT (GPT-3.5).
• Construct prompts including QAP (question-answer pair) and CoT explanations to guide inference.
• Evaluate on SemEval-2016, VAST (zero-shot), and P-Stance datasets using macro-F1 and Favg metrics.
• Benchmark against a range of stance detection baselines (e.g., Bicond, CrossNet, SEKT, MemNet, AOA, TAN, ASGCN, Bert_spc, Bert-GCN, PT-HCL).
• Analyze factors affecting CoT performance, including target biases, number and type of QAPs, and granularity of the stance task.

Experimental results

Research questions

• RQ1 Can CoT prompting with ChatGPT achieve state-of-the-art or competitive stance detection performance without training?
• RQ2 How do different CoT prompting strategies (DQA vs. StSQA) compare in zero-shot and one-shot settings across standard datasets?
• RQ3 What limitations and biases affect CoT-based stance detection on social media (e.g., target bias, QAP selection, task granularity)?

Key findings

• CoT prompting with ChatGPT can achieve state-of-the-art or comparable performance on stance detection without training across multiple datasets.
• StSQA attains strong zero-shot performance and often outperforms competitive baselines in zero-shot settings.
• ChatGPT shows target biases that can degrade performance on certain topics, indicating prompting strategies and target selection need careful design.
• The number and type of QAPs significantly impact performance, with overly specific prompts sometimes reducing accuracy.
• Fine-grained stance definitions (multiple perspectives on a target) pose challenges for direct polarity classification and warrant further research.