[Paper Review] A Logically Consistent Chain-of-Thought Approach for Stance Detection

Bowen Zhang, Daijun Ding+2 more|arXiv (Cornell University)|Dec 26, 2023

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes Logically Consistent Chain-of-Thought (LC-CoT), a three-stage LLM-based approach for zero-shot stance detection that ensures knowledge relevance and logical consistency. By first assessing knowledge necessity, then retrieving targeted background knowledge via API calls, and finally applying if-then logical templates guided by manual exemplars, LC-CoT achieves state-of-the-art performance without any labeled training data, outperforming supervised and CoT-based methods on SEM16 and VAST datasets.

ABSTRACT

Zero-shot stance detection (ZSSD) aims to detect stances toward unseen targets. Incorporating background knowledge to enhance transferability between seen and unseen targets constitutes the primary approach of ZSSD. However, these methods often struggle with a knowledge-task disconnect and lack logical consistency in their predictions. To address these issues, we introduce a novel approach named Logically Consistent Chain-of-Thought (LC-CoT) for ZSSD, which improves stance detection by ensuring relevant and logically sound knowledge extraction. LC-CoT employs a three-step process. Initially, it assesses whether supplementary external knowledge is necessary. Subsequently, it uses API calls to retrieve this knowledge, which can be processed by a separate LLM. Finally, a manual exemplar guides the LLM to infer stance categories, using an if-then logical structure to maintain relevance and logical coherence. This structured approach to eliciting background knowledge enhances the model's capability, outperforming traditional supervised methods without relying on labeled data.

Motivation & Objective

• To address knowledge-task disconnect and lack of logical consistency in zero-shot stance detection (ZSSD) using large language models (LLMs).
• To improve transferability to unseen targets by ensuring retrieved background knowledge is both relevant and logically coherent.
• To develop a method that enhances ZSSD performance without requiring labeled training data.
• To integrate structured reasoning (if-then logic) into knowledge elicitation for more reliable stance prediction.

Proposed method

• The method first uses an LLM to determine whether external knowledge is needed for stance prediction based on input text and target.
• If knowledge is required, the LLM generates API calls to retrieve relevant background information from an external LLM or knowledge base.
• A separate LLM processes the retrieved knowledge to ensure contextual relevance and factual consistency.
• The final inference stage uses a manually curated exemplar with if-then logical structures to guide the LLM in combining input text and background knowledge for stance classification.
• The entire process is structured as a chain-of-thought (CoT) with explicit logical scaffolding to maintain coherence and reduce hallucination.
• The approach avoids fine-tuning and relies solely on prompt engineering and API-based knowledge retrieval.

Experimental results

Research questions

• RQ1 Can a logically structured chain-of-thought approach improve zero-shot stance detection by reducing knowledge-task disconnect?
• RQ2 How does integrating if-then logical reasoning in knowledge application affect prediction consistency and accuracy in ZSSD?
• RQ3 To what extent can an LLM-based method outperform supervised models in ZSSD without any labeled training data?
• RQ4 Does API-driven retrieval of targeted knowledge enhance generalization to unseen targets compared to end-to-end prompting?

Key findings

• LC-CoT achieved a macro F1 score of 82.9 on the SEM16 dataset, outperforming the best baseline (TarBK) by 19.5% on average across three targets.
• On the VAST dataset, LC-CoT achieved a macro F1 of 72.5, surpassing KASD-ChatGPT by 5.5% and GPT-CoT by 10.6%.
• In cross-target stance detection, LC-CoT improved F1 scores by 18.1% on average over TPDG and 16.3% over PT-HCL, demonstrating strong generalization.
• LC-CoT achieved state-of-the-art performance on both SEM16 and VAST datasets without requiring any labeled training data.
• The method significantly reduced knowledge-task disconnect and improved logical consistency, as evidenced by higher F1 scores and stable performance across diverse target domains.
• The use of if-then logical templates in the final reasoning step enhanced prediction reliability and reduced contradictory or irrelevant knowledge usage.