[Paper Review] TPE: Towards Better Compositional Reasoning over Conceptual Tools with Multi-persona Collaboration

Hongru Wang, Huimin Wang+7 more|arXiv (Cornell University)|Sep 28, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes TPE, a multi-persona collaboration framework that enhances compositional reasoning in dialogue systems by decomposing response generation into three roles: Thinker (analyzes user context), Planner (plans conceptual tool usage), and Executor (executes and composes responses). TPE improves reasoning over conceptual tools—such as knowledge sources and tutoring strategies—demonstrating state-of-the-art performance on multi-source (FoCus) and multi-strategy (CIMA, PsyQA) dialogue benchmarks with reduced token redundancy and enhanced explainability.

ABSTRACT

Large language models (LLMs) have demonstrated exceptional performance in planning the use of various functional tools, such as calculators and retrievers, particularly in question-answering tasks. In this paper, we expand the definition of these tools, centering on conceptual tools within the context of dialogue systems. A conceptual tool specifies a cognitive concept that aids systematic or investigative thought. These conceptual tools play important roles in practice, such as multiple psychological or tutoring strategies being dynamically applied in a single turn to compose helpful responses. To further enhance the reasoning and planning capability of LLMs with these conceptual tools, we introduce a multi-persona collaboration framework: Think-Plan-Execute (TPE). This framework decouples the response generation process into three distinct roles: Thinker, Planner, and Executor. Specifically, the Thinker analyzes the internal status exhibited in the dialogue context, such as user emotions and preferences, to formulate a global guideline. The Planner then generates executable plans to call different conceptual tools (e.g., sources or strategies), while the Executor compiles all intermediate results into a coherent response. This structured approach not only enhances the explainability and controllability of responses but also reduces token redundancy. We demonstrate the effectiveness of TPE across various dialogue response generation tasks, including multi-source (FoCus) and multi-strategy interactions (CIMA and PsyQA). This reveals its potential to handle real-world dialogue interactions that require more complicated tool learning beyond just functional tools. The full code and data will be released for reproduction.

Motivation & Objective

• To address the limitation of existing LLM-based dialogue systems that primarily handle functional tools (e.g., APIs, retrievers) but underutilize conceptual tools—mental constructs like psychological strategies or knowledge sources that support systematic thought.
• To improve the reasoning and planning capabilities of LLMs in complex dialogue scenarios requiring dynamic, multi-step integration of diverse conceptual tools.
• To enhance response explainability, controllability, and personalization by structuring the reasoning process into distinct, role-specific phases.
• To demonstrate the effectiveness of this framework across diverse dialogue tasks involving multiple sources and strategies, beyond traditional functional tool use.

Proposed method

• The TPE framework introduces three distinct personas: Thinker, Planner, and Executor, each responsible for a specific stage of response generation.
• The Thinker analyzes the dialogue context to infer user emotions, preferences, and intent, generating a global guideline for response generation.
• The Planner formulates a sequence of executable actions to invoke conceptual tools—such as document retrievers or psychological strategies—based on the Thinker’s output.
• The Executor integrates intermediate results from multiple conceptual tools into a coherent, natural language response.
• Each component is prompted with role-specific instructions, ensuring structured, modular reasoning and reducing token redundancy.
• The framework is applied to multi-source (FoCus) and multi-strategy (CIMA, PsyQA) dialogue tasks, with zero-shot prompting and no fine-tuning.

Experimental results

Research questions

• RQ1 Can a structured multi-persona collaboration framework improve compositional reasoning in LLMs when handling conceptual tools such as knowledge sources and psychological strategies?
• RQ2 How does separating reasoning into Thinker, Planner, and Executor roles affect response quality, explainability, and controllability in complex dialogue systems?
• RQ3 To what extent can TPE generalize across diverse dialogue tasks requiring dynamic integration of multiple conceptual tools without task-specific fine-tuning?
• RQ4 Does TPE reduce token redundancy compared to standard autoregressive generation in multi-tool dialogue scenarios?

Key findings

• TPE achieves state-of-the-art performance on the FoCus benchmark for multi-source dialogue, demonstrating superior reasoning over multiple knowledge sources.
• On the CIMA benchmark for multi-strategy tutoring dialogue, TPE outperforms strong baselines by effectively composing educational strategies such as hints, questions, and corrections in a contextually appropriate sequence.
• The framework significantly reduces token redundancy by decoupling planning from execution, leading to more concise and focused responses.
• TPE enhances response explainability and controllability by making the reasoning process modular and traceable through distinct persona roles.
• The zero-shot application of TPE across diverse dialogue tasks shows strong generalization, indicating its robustness beyond fine-tuned models.
• The ablation study confirms that each component—Thinker, Planner, Executor—contributes meaningfully to the final performance, validating the design of the multi-persona collaboration.