[Paper Review] Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitation

Yu Wang, Shiwan Zhao+7 more|arXiv (Cornell University)|Sep 5, 2024

Big Data and Business Intelligence4 citations

TL;DR

This paper proposes Strategic Chain-of-Thought (SCoT), a single-prompt, two-stage method that first elicits an optimal problem-solving strategy before generating a Chain-of-Thought (CoT) reasoning path, significantly improving reasoning accuracy in LLMs. Using Llama3-8b, SCoT achieves a 21.05% improvement on GSM8K and 24.13% on Tracking_Objects, outperforming standard CoT without requiring multiple queries or external knowledge sources.

ABSTRACT

The Chain-of-Thought (CoT) paradigm has emerged as a critical approach for enhancing the reasoning capabilities of large language models (LLMs). However, despite their widespread adoption and success, CoT methods often exhibit instability due to their inability to consistently ensure the quality of generated reasoning paths, leading to sub-optimal reasoning performance. To address this challenge, we propose the extbf{Strategic Chain-of-Thought} (SCoT), a novel methodology designed to refine LLM performance by integrating strategic knowledge prior to generating intermediate reasoning steps. SCoT employs a two-stage approach within a single prompt: first eliciting an effective problem-solving strategy, which is then used to guide the generation of high-quality CoT paths and final answers. Our experiments across eight challenging reasoning datasets demonstrate significant improvements, including a 21.05\% increase on the GSM8K dataset and 24.13\% on the Tracking\_Objects dataset, respectively, using the Llama3-8b model. Additionally, we extend the SCoT framework to develop a few-shot method with automatically matched demonstrations, yielding even stronger results. These findings underscore the efficacy of SCoT, highlighting its potential to substantially enhance LLM performance in complex reasoning tasks.

Motivation & Objective

• To address the instability and variability in reasoning path quality produced by standard Chain-of-Thought (CoT) methods in LLMs.
• To improve reasoning accuracy without relying on multi-query methods or external knowledge sources.
• To develop a strategy-elicitation framework that guides high-quality CoT path generation in a single prompt.
• To extend SCoT into a few-shot method using strategy-based demonstration matching for enhanced performance.
• To evaluate the feasibility of automatically generating SCoT prompt templates using LLMs.

Proposed method

• SCoT employs a two-stage prompting process: first, it elicits an effective problem-solving strategy within a single prompt.
• The elicited strategy is then used to guide the generation of a high-quality CoT reasoning path and final answer.
• The method integrates role assignment, structured workflows, and markdown formatting to improve prompt effectiveness.
• A few-shot extension uses strategic knowledge to automatically match relevant demonstrations from a predefined corpus.
• The approach avoids multi-query inference and external knowledge, reducing computational cost and operational overhead.
• Automatic prompt generation is evaluated by using a large language model (Qwen2-72B) to generate SCoT templates based on the SCoT concept.

Experimental results

Research questions

• RQ1 Can eliciting a problem-solving strategy before CoT generation improve reasoning accuracy in LLMs?
• RQ2 How does SCoT compare to standard CoT and other multi-query or knowledge-augmented methods in terms of performance and efficiency?
• RQ3 Can SCoT be effectively extended to a few-shot setting using strategy-based demonstration selection?
• RQ4 Is it feasible to automatically generate effective SCoT prompt templates using a large language model?
• RQ5 What impact do prompt components—such as role, workflow, structure, and number of demonstrations—have on SCoT’s accuracy?

Key findings

• SCoT achieved a 21.05% absolute improvement on the GSM8K dataset using the Llama3-8b model compared to standard CoT.
• On the Tracking_Objects dataset, SCoT improved performance by 24.13% over CoT, demonstrating strong gains in complex reasoning tasks.
• The few-shot extension of SCoT further enhanced performance by leveraging strategy-based demonstration matching from a curated corpus.
• Token length in SCoT outputs was on average 1.5 times that of CoT, with a range of 1.03 to 1.8 times, indicating manageable efficiency overhead.
• Automatic generation of SCoT prompts using Qwen2-72B yielded 31.89% accuracy on AQuA, surpassing 0-shot CoT (29.13%) and approaching manually crafted SCoT (33.60%).
• The method reduced repetitive output patterns on GSM8K, particularly in CoT 0-shot, where model output was prone to redundant token generation under length constraints.