[Paper Review] Reasoning Implicit Sentiment with Chain-of-Thought Prompting

Hao Fei, Bobo Li+4 more|arXiv (Cornell University)|May 18, 2023

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes ThoR, a three-hop chain-of-thought prompting framework that enhances implicit sentiment analysis (ISA) by sequentially reasoning through fine-grained aspect, underlying opinion, and final sentiment polarity. By leveraging large language models (LLMs) with structured CoT prompting and self-consistency, ThoR achieves a 51.10% F1 improvement over baselines on zero-shot ISA using GPT-3 (175B), significantly outperforming prior methods.

ABSTRACT

While sentiment analysis systems try to determine the sentiment polarities of given targets based on the key opinion expressions in input texts, in implicit sentiment analysis (ISA) the opinion cues come in an implicit and obscure manner. Thus detecting implicit sentiment requires the common-sense and multi-hop reasoning ability to infer the latent intent of opinion. Inspired by the recent chain-of-thought (CoT) idea, in this work we introduce a Three-hop Reasoning (THOR) CoT framework to mimic the human-like reasoning process for ISA. We design a three-step prompting principle for THOR to step-by-step induce the implicit aspect, opinion, and finally the sentiment polarity. Our THOR+Flan-T5 (11B) pushes the state-of-the-art (SoTA) by over 6% F1 on supervised setup. More strikingly, THOR+GPT3 (175B) boosts the SoTA by over 50% F1 on zero-shot setting. Our code is open at https://github.com/scofield7419/THOR-ISA.

Motivation & Objective

• To address the challenge of implicit sentiment analysis (ISA), where opinion cues are absent or obscured, requiring common-sense and multi-hop reasoning.
• To improve sentiment prediction accuracy in ISA by mimicking human-like reasoning through step-by-step inference.
• To extend the chain-of-thought (CoT) prompting paradigm to the sentiment analysis domain, particularly for implicit sentiment detection.
• To demonstrate that structured, incremental reasoning with LLMs significantly outperforms direct prediction in zero-shot and supervised setups.

Proposed method

• Design a three-step prompting framework that first infers the fine-grained aspect of the target, then the latent opinion, and finally the sentiment polarity.
• Implement a self-consistency mechanism to improve reasoning reliability by selecting candidate answers with high voting consistency across multiple reasoning paths.
• Introduce a reasoning revising method for supervised fine-tuning, where intermediate reasoning outputs are used as input to predict final labels, guided by gold annotations.
• Utilize large language models (LLMs) such as Flan-T5 (11B) and GPT-3 (175B) as the backbone, leveraging their world knowledge and reasoning capabilities.
• Apply CoT prompting to elicit intermediate reasoning steps, transforming the model into a multi-hop reasoner rather than a direct label predictor.
• Integrate the framework into both zero-shot and supervised fine-tuning settings to evaluate generalization and performance gains.

Experimental results

Research questions

• RQ1 Can chain-of-thought prompting effectively improve implicit sentiment analysis by enabling step-by-step reasoning in LLMs?
• RQ2 How does structured three-hop reasoning—aspect → opinion → polarity—enhance sentiment prediction compared to direct classification?
• RQ3 To what extent does model scale amplify the performance gain of the ThoR framework in zero-shot and supervised settings?
• RQ4 How does self-consistency in reasoning paths improve robustness and reduce error rates in implicit sentiment detection?
• RQ5 What are the main failure modes of ThoR, and how do they differ between zero-shot and fine-tuned setups?

Key findings

• ThoR + Flan-T5 (11B) achieves a 6%+ improvement in F1 score over the best baseline in the supervised setting.
• ThoR + GPT-3 (175B) achieves a 51.10% relative F1 improvement over the baseline in the zero-shot setting, demonstrating strong generalization.
• The error rate for zero-shot ThoR with Flan-T5-11B drops from 48.27% to 12.79% when fine-tuned, indicating the benefit of supervision on reasoning quality.
• Failure analysis reveals that unsupervised GPT-3 errors stem mainly from noisy data annotations, while fine-tuned T5 models are more prone to learning spurious correlations.
• The performance gain of ThoR is more pronounced in larger models, confirming that its effectiveness is tied to the emergent reasoning capabilities of large-scale LLMs.
• ThoR is the first successful application of chain-of-thought prompting to implicit sentiment analysis, establishing a new SOTA in both zero-shot and supervised setups.