[Paper Review] Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs

Hongru Wang, Rui Wang+6 more|arXiv (Cornell University)|May 19, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Cue-CoT, a chain-of-thought prompting method that enhances LLM-based dialogue systems by explicitly reasoning over linguistic cues—personality, emotion, and psychology—exhibited in user dialogue contexts. By decomposing response generation into intermediate reasoning steps, Cue-CoT improves response helpfulness and acceptability across 6 multilingual datasets, with M-Cue CoT showing superior robustness and performance over standard prompting in both zero-shot and one-shot settings.

ABSTRACT

Large Language Models (LLMs), such as exttt{ChatGPT}, greatly empower dialogue systems with strong language understanding and generation capabilities. However, most of the previous works prompt the LLMs to directly generate a response based on the dialogue context, overlooking the underlying linguistic cues about the user status exhibited in the context. Such in-depth dialogue scenarios are challenging for existing LLMs to figure out the user's hidden needs and respond satisfactorily through a single-step inference. To this end, we propose a novel linguistic cue-based chain-of-thoughts ( extit{Cue}-CoT), which enhances the LLMs inference with an intermediate reasoning step to find cues exhibited in the dialogue, aiming to provide a more personalized and engaging response. To evaluate the approach, we build a benchmark with in-depth dialogue questions, consisting of 6 datasets in both Chinese and English, targeting 3 major linguistic cues during the conversation: extit{personality}, extit{emotion}, and extit{psychology}. We conduct extensive experiments on the proposed benchmark with 5 LLMs under both zero-shot and one-shot settings. Empirical results demonstrate our proposed extit{Cue}-CoT method outperforms standard prompting methods in terms of both extit{helpfulness} and extit{acceptability} on all datasets.

Motivation & Objective

• To address the limitation of standard prompting in generating personalized, context-aware responses for in-depth dialogue questions.
• To investigate how linguistic cues—personality, emotion, and psychology—can be leveraged to improve LLM reasoning in dialogue systems.
• To develop a novel chain-of-thought prompting framework that explicitly models user status through intermediate reasoning steps.
• To construct a comprehensive benchmark with 6 multilingual datasets to evaluate user-status-aware dialogue generation.
• To explore demonstration selection strategies based on intermediate reasoning for few-shot settings.

Proposed method

• Proposes two variants of Cue-CoT: O-Cue CoT (one-step reasoning with final response) and M-Cue CoT (step-by-step reasoning over user status cues).
• Designs prompts that guide LLMs to first infer personality, emotion, and psychological status from dialogue context before generating a response.
• Introduces a novel demonstration selection strategy in one-shot settings that uses intermediate reasoning results as selection criteria.
• Builds a benchmark with 6 datasets (3 in Chinese, 3 in English) covering 7,300 dialogues, annotated for personality, emotion, and psychological cues.
• Employs zero-shot and one-shot prompting with 5 LLMs to evaluate performance across diverse models and languages.
• Uses automatic and human evaluation to measure response helpfulness and acceptability, with no direct evaluation of intermediate reasoning correctness due to ambiguity and one-to-many mapping.

Experimental results

Research questions

• RQ1 Can reasoning over linguistic cues—personality, emotion, and psychology—improve the quality of LLM-generated dialogue responses?
• RQ2 How does Cue-CoT compare to standard prompting in terms of response helpfulness and acceptability across diverse dialogue scenarios?
• RQ3 Does M-Cue CoT outperform O-Cue CoT in reasoning robustness and response quality?
• RQ4 Can intermediate reasoning results be used effectively to guide demonstration selection in few-shot settings?
• RQ5 How do different LLMs perform across multilingual in-depth dialogue benchmarks when using Cue-CoT?

Key findings

• Cue-CoT significantly outperforms standard prompting in both helpfulness and acceptability across all 6 datasets under zero-shot and one-shot settings.
• M-Cue CoT demonstrates superior robustness and reasoning performance compared to O-Cue CoT across all datasets and LLMs.
• The proposed demonstration selection strategy based on intermediate reasoning results achieves better performance than random or top-1 selection in one-shot settings.
• Empirical results show that LLMs are sensitive to prompt design, and optimal prompts are crucial for performance.
• Most LLMs perform better in English (Area 2) or Chinese (Area 3) than in both languages simultaneously (Area 4), indicating a need for better multilingual alignment.
• The study highlights the challenge of evaluating intermediate reasoning due to the one-to-many nature of dialogue response generation and the possibility of correct answers via incorrect reasoning.