[Paper Review] Graph of Thoughts: Solving Elaborate Problems with Large Language Models

Maciej Besta, Nils Blach+9 more|arXiv (Cornell University)|Aug 18, 2023

Topic Modeling73 references35 citations

TL;DR

GoT models LLM reasoning as an arbitrary graph of thoughts, enabling aggregation and feedback to outperform Tree of Thoughts while reducing costs. It supports new prompting transformations and demonstrates strong results on tasks like sorting.

ABSTRACT

We introduce Graph of Thoughts (GoT): a framework that advances prompting capabilities in large language models (LLMs) beyond those offered by paradigms such as Chain-of-Thought or Tree of Thoughts (ToT). The key idea and primary advantage of GoT is the ability to model the information generated by an LLM as an arbitrary graph, where units of information ("LLM thoughts") are vertices, and edges correspond to dependencies between these vertices. This approach enables combining arbitrary LLM thoughts into synergistic outcomes, distilling the essence of whole networks of thoughts, or enhancing thoughts using feedback loops. We illustrate that GoT offers advantages over state of the art on different tasks, for example increasing the quality of sorting by 62% over ToT, while simultaneously reducing costs by >31%. We ensure that GoT is extensible with new thought transformations and thus can be used to spearhead new prompting schemes. This work brings the LLM reasoning closer to human thinking or brain mechanisms such as recurrence, both of which form complex networks.

Motivation & Objective

• Motivate the need for more powerful prompting beyond CoT and ToT by enabling graph-based reasoning in LLMs.
• Propose Graph of Thoughts (GoT) as a modular framework to represent and transform LLM thoughts as a graph.
• Demonstrate how graph-enabled transformations improve task performance and enable new prompting schemes.
• Evaluate GoT against state-of-the-art prompting schemes on tasks like sorting, set operations, keyword counting, and document merging.

Proposed method

• Model LLM reasoning as a directed graph G=(V,E) where thoughts are vertices and dependencies are edges.
• Introduce graph-enabled transformations such as aggregation, refinement, and generation to modify G and produce new thoughts.
• Define scoring E(v,G,p_theta) and ranking R(G,p_theta,h) to evaluate and select top thoughts.
• Provide a modular system architecture with Prompter, Parser, Scoring, and Controller, plus GoO (Graph of Operations) and GRS (Graph Reasoning State) for execution planning and state tracking.
• Extend GoT with heterogeneous graph representations and explicit removal of thoughts to manage context size.

Experimental results

Research questions

• RQ1 How can prompting schemes transition from linear chains or trees to arbitrary graphs of thoughts to improve problem solving?
• RQ2 What graph transformations and architectures maximize solution quality while minimizing LLM inference cost?
• RQ3 How does GoT compare to CoT, CoT-SC, and ToT in tasks like sorting, set operations, keyword counting, and document merging?
• RQ4 What is the impact of graph structure on latency and information volume in LLM reasoning?
• RQ5 Can GoT be extended with new prompting schemes and different LLMs without model updates?

Key findings

• GoT improves sorting quality by approximately 62% over Tree of Thoughts while reducing cost by more than 31%.
• GoT demonstrates higher-quality outcomes across tasks like sorting, set intersection, keyword counting, and document merging compared to baselines.
• GoT achieves a favorable latency-volume tradeoff with latency log_k N and high information volume N, unlike CoT (high volume, high latency), CoT-SC (reduced latency and volume), or ToT (low latency and low volume).
• The framework enables aggregation and other graph-enabled transformations, allowing solving subtasks independently and merging results for final solutions.
• GoT is extensible with new thought transformations and prompts, enabling rapid prototyping across models such as GPT-3.5, GPT-4, and Llama-2.