[Paper Review] Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models

Anni Zou, Zhuosheng Zhang+2 more|arXiv (Cornell University)|Oct 10, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes GeM-CoT, a generalizable chain-of-thought prompting framework for mixed-task scenarios where input question types are unknown. It uses type classification and diversity-based demonstration retrieval or construction from a dynamically updated pool, achieving state-of-the-art performance and strong generalization across 33 reasoning tasks, including 10 public datasets and 23 BBH tasks.

ABSTRACT

Large language models (LLMs) have unveiled remarkable reasoning capabilities by exploiting chain-of-thought (CoT) prompting, which generates intermediate reasoning chains to serve as the rationale for deriving the answer. However, current CoT methods either simply employ general prompts such as Let's think step by step, or heavily rely on pre-defined task-specific demonstrations to attain preferable performances, thereby engendering an inescapable gap between performance and generalization. To bridge this gap, we propose GeM-CoT, a Generalizable CoT prompting mechanism in Mixed-task scenarios where the type of input questions is unknown. GeM-CoT first categorizes the question type and subsequently samples or constructs demonstrations from the corresponding data pool in an automatic pattern. With this technical design, GeM-CoT simultaneously enjoys superior generalization capabilities and remarkable performances on 10 public reasoning tasks and 23 BBH tasks.

Motivation & Objective

• Address the performance-generalization gap in chain-of-thought prompting under realistic mixed-task scenarios where question types are unknown and dynamically mixed.
• Develop a prompting mechanism that maintains high performance without relying on task-specific demonstrations or manual labeling.
• Enable automatic, scalable, and cost-free demonstration retrieval and construction to support long-term adaptability in open-ended environments.
• Bridge the trade-off between zero-shot generalization and few-shot performance by unifying both through a unified, self-updating framework.
• Demonstrate robustness and generalization across diverse reasoning tasks, including arithmetic, commonsense, and symbolic reasoning.

Proposed method

• First, classify the input question into a task type using semantic similarity matching against a pre-constructed demo pool.
• For successfully matched questions, retrieve diverse, high-quality demonstrations from the corresponding task-specific data pool.
• For unmatched questions, perform zero-shot reasoning and store the input-question-answer triplet in a data cache.
• Apply density-based clustering to grouped questions in the cache to identify clusters meeting quality criteria.
• Automatically construct diverse, task-relevant demonstrations from each qualified cluster and add them to the demo pool for future use.
• Continuously update the demo pool and cache to ensure long-term adaptability and improved generalization over time.

Figure 1: Comparison of conventional single-task scenarios and our concerned setting: mixed-task scenarios . There are three major characteristics of mixed-task scenarios: (i) the type of any incoming question is unknown; (ii) the input data comes from a set of mixed tasks; (iii) the questions come

Experimental results

Research questions

• RQ1 Can a chain-of-thought prompting mechanism achieve high performance while maintaining strong generalization in mixed-task scenarios with unknown question types?
• RQ2 How effective is diversity-based demonstration selection compared to similarity-based or random sampling in few-shot CoT prompting?
• RQ3 To what extent does type matching based on semantic similarity outperform LLM-based classifiers in terms of cost, accuracy, and generalization?
• RQ4 What threshold value in the type matching module optimally balances precision and recall across diverse reasoning tasks?
• RQ5 Can automatic demonstration construction from clustered data improve both performance and generalization in zero-shot and few-shot settings?

Key findings

• GeM-CoT achieves state-of-the-art performance on 10 public reasoning benchmarks, including 93.7% accuracy on AddSub and 100% on Coin.
• The method outperforms both similarity-based and random demonstration selection strategies, with a 51.9% accuracy on AQuA and 63.5% on Strategy.
• The type matching module based on semantic similarity achieves comparable performance to an LLM-based classifier but without API costs or prompt engineering.
• Using a threshold of 0.35 in the type matching module yields optimal F1 and accuracy across all tasks, as validated by analysis on 1,200 test questions.
• GeM-CoT demonstrates strong generalization, with performance improving steadily from batch 2 onward, indicating growing adaptability with more data.
• The approach successfully unifies performance and generalization, achieving high accuracy across 23 BBH tasks while maintaining broad applicability in open-ended, mixed-task environments.

Figure 2: Overview of our proposed GeM-CoT mechanism. GeM-CoT first routes the input question to different paths ( Type Matching ): i) path matched $\rightarrow$ : For a successful match, it fetches demonstrations from the demo pool ( Demo Acquisition ) and performs a final inference ( Answer Deriva