[Paper Review] Diversity of Thought Improves Reasoning Abilities of LLMs

Ranjita Naik, Varun Chandrasekaran+3 more|arXiv (Cornell University)|Oct 11, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes Div-Se and IDiv-Se, methods that enhance large language model (LLM) reasoning by generating diverse prompting strategies through LLM feedback, then ensembling responses across multiple inference calls (Div-Se) or within a single call (IDiv-Se). The approach improves accuracy without modifying decoding, achieving state-of-the-art results—up to 29.6 percentage points higher on the 4/5 Blocksworld task—while advancing the accuracy-cost Pareto frontier on multiple reasoning benchmarks using GPT-3.5 and GPT-4.

ABSTRACT

Large language models (LLMs) are documented to struggle in settings that require complex reasoning. Nevertheless, instructing the model to break down the problem into smaller reasoning steps, or ensembling various generations through modifying decoding steps boosts performance. However, these methods assume that the input prompt is fixed and expect the decoding strategies to introduce the diversity needed for ensembling. In this work, we discuss how one can create and leverage variations of the input prompt as a means of diversity of thought. We propose a method that automatically improves prompt diversity by soliciting feedback from the LLM to ideate approaches that are apt for the problem. We then ensemble the diverse prompts in our method DIVSE (DIVerse reasoning path Self-Ensemble) across multiple inference calls, or use diverse approaches within a single inference call; we call the latter IDIV-SE (In-call DIVerse reasoning path Self-Ensemble). Apart from our approaches outperforming prior work, DIV-SE(in particular) advances state-of-the-art performance on the challenging planning and graph coloring benchmarks. Our results improve the Pareto frontier of the accuracy-cost trade-off.

Motivation & Objective

• To address the limitation of fixed prompts in few-shot and zero-shot CoT prompting, where performance gains rely solely on decoding stochasticity.
• To improve reasoning performance in LLMs without modifying decoding processes, especially in commercial or black-box deployment settings.
• To explore whether diversity at the thought level—i.e., different problem-solving approaches—can enhance ensemble accuracy more effectively than token-level stochasticity.
• To reduce inference cost while maintaining or improving performance by enabling multiple diverse prompts to be processed in a single call via IDiv-Se.
• To establish diversity of thought as a principled prompting strategy that advances the Pareto frontier of accuracy versus cost in LLM reasoning.

Proposed method

• The method solicits LLM feedback to generate multiple high-level problem-solving approaches (e.g., working backwards, visualization, elimination) for a given reasoning task.
• Each identified approach is used to style-transfer exemplar reasoning steps from prior work (e.g., Chain-of-Thought) into a new prompt format aligned with that approach.
• Div-Se performs multiple independent inference calls, each using a different diverse prompt, and aggregates results via majority voting.
• IDiv-Se combines n diverse prompts into a single input prompt, generating n reasoning paths in one inference call, then aggregates outputs using majority voting.
• The approach leverages the model’s ability to suggest varied reasoning strategies, ensuring diversity at the thought level rather than at the token level.
• Aggregation is performed via majority voting or meta-reasoning techniques to improve final output reliability.

Figure 1: Diversity of Thought enhances the inference cost and accuracy trade-off. We compare DIV-SE and IDIV-SE with SC (Wang et al., 2023 ) and CoT (Wei et al., 2022 ) across three benchmarks. Panels show (i) AQUA-RAT on GPT-3.5 in few-shot-CoT setting, and (ii) Blocksworld 3 and 4/5 on GPT-4 in z

Experimental results

Research questions

• RQ1 Can soliciting diverse problem-solving approaches from an LLM improve reasoning performance beyond standard CoT and self-consistency methods?
• RQ2 Does diversity at the thought level—such as different strategies like working backwards or visualization—yield better ensemble accuracy than diversity at the token level?
• RQ3 Can a single-inference-call method (IDiv-Se) achieve performance comparable to multi-call ensembling (Div-Se) while reducing cost?
• RQ4 To what extent does prompt diversity, generated via LLM feedback, advance the accuracy-cost Pareto frontier in reasoning benchmarks?
• RQ5 Can diverse prompting strategies independently outperform zero-shot CoT, even without ensemble aggregation?

Key findings

• Div-Se and IDiv-Se outperform both Chain-of-Thought (CoT) and self-consistency (SC) prompting on GPT-3.5 and GPT-4 across multiple reasoning benchmarks, including AQUA-RAT and Blocksworld.
• On the challenging 4/5 Blocksworld task, Div-Se achieves a 29.6 percentage point improvement over the prior state-of-the-art, exceeding the highest reported accuracy by that margin.
• On the AQUA-RAT benchmark, IDiv-Se improves performance by 16.52 percentage points over CoT in the few-shot setting with GPT-3.5.
• IDiv-Se achieves comparable accuracy to Div-Se at significantly lower inference cost, demonstrating the viability of in-call ensemble inference.
• Even without ensemble aggregation, diverse prompts derived from LLM-suggested approaches outperform standard zero-shot CoT, indicating intrinsic value in prompt diversity.
• Meta-reasoning aggregation further improves results, suggesting that reasoning paths contain rich information exploitable by post-hoc techniques.

Figure 2: Diversity of Thought . This illustration depicts CoT and IDiv-Se prompting strategies. Notice that both have a single example. However, IDiv-Se presents more diversity in terms of reasoning paths. This enables it to generate diverse completions, yielding more accurate responses.