[논문 리뷰] Demystifying Chains, Trees, and Graphs of Thoughts

Maciej Besta, Florim Memedi외 14명|arXiv (Cornell University)|2024. 01. 25.

Topic Modeling인용 수 6

한 줄 요약

본 논문은 LLM의 구조 강화 프롬프트에 대한 일반적 설계도와 분류체계를 제시하고, 추론 토폴로지(연쇄(chain), 트리(tree), 그래프(graph))를 도입하며 기존 스킴(CoT, ToT, GoT)과 그 트레이드오프를 분석한다.

ABSTRACT

The field of natural language processing (NLP) has witnessed significant progress in recent years, with a notable focus on improving large language models' (LLM) performance through innovative prompting techniques. Among these, prompt engineering coupled with structures has emerged as a promising paradigm, with designs such as Chain-of-Thought, Tree of Thoughts, or Graph of Thoughts, in which the overall LLM reasoning is guided by a structure such as a graph. As illustrated with numerous examples, this paradigm significantly enhances the LLM's capability to solve numerous tasks, ranging from logical or mathematical reasoning to planning or creative writing. To facilitate the understanding of this growing field and pave the way for future developments, we devise a general blueprint for effective and efficient LLM reasoning schemes. For this, we conduct an in-depth analysis of the prompt execution pipeline, clarifying and clearly defining different concepts. We then build the first taxonomy of structure-enhanced LLM reasoning schemes. We focus on identifying fundamental classes of harnessed structures, and we analyze the representations of these structures, algorithms executed with these structures, and many others. We refer to these structures as reasoning topologies, because their representation becomes to a degree spatial, as they are contained within the LLM context. Our study compares existing prompting schemes using the proposed taxonomy, discussing how certain design choices lead to different patterns in performance and cost. We also outline theoretical underpinnings, relationships between prompting and other parts of the LLM ecosystem such as knowledge bases, and the associated research challenges. Our work will help to advance future prompt engineering techniques.

연구 동기 및 목표

• crystallize fundamental building blocks and concepts in the prompt execution pipeline
• develop a taxonomy of structure-enhanced LLM reasoning schemes based on reasoning topologies
• survey and compare existing prompting schemes through the proposed taxonomy
• analyze how topology choices affect performance, latency, and cost
• outline research challenges and future directions in prompt engineering “topologies”

제안 방법

• analyze the prompt execution pipeline and define core components (f_pre, LLM^X, f_post, f_c, f'_c)
• define and formalize reasoning topologies as graphs of thoughts (nodes/edges) within and across prompts
• establish a blueprint with topology class, topology scope, representation, derivation, reasoning schedule, and pipeline integration
• classify and compare existing schemes (CoT, ToT, GoT, etc.) using the taxonomy in terms of representation, derivation, and scheduling
• provide a framework to analyze trade-offs in accuracy, latency, and cost across different topologies
• discuss theoretical underpinnings and relationships to knowledge bases and broader LLM ecosystems

실험 결과

연구 질문

• RQ1 What are the fundamental building blocks and concepts of the general prompt execution pipeline?
• RQ2 How can we define and categorize reasoning topologies (chains, trees, graphs) within LLM prompting?
• RQ3 How do representation, derivation, and scheduling choices of topologies affect performance and cost?
• RQ4 What are the open challenges and research directions for structure-enhanced prompting topologies?

주요 결과

• First taxonomy of structure-enhanced LLM reasoning schemes is proposed and applied to CoT, ToT, GoT, and related prompting schemes.
• Reasoning topologies are modeled as graphs of thoughts with nodes (thoughts) and edges (dependencies).
• A formal blueprint is provided to analyze and instantiate topology-based reasoning, including representation, derivation, and scheduling across single- or multi-prompt settings.
• Comparisons show how different topology classes and schedules impact accuracy, latency, and cost, highlighting trade-offs.
• The work connects prompting design with broader AI pipeline components (retrieval, tools, knowledge bases) and outlines future research directions.