[Paper Review] AFlow: Automating Agentic Workflow Generation

Jiayi Zhang, Jianhai Xiang+12 more|arXiv (Cornell University)|Oct 14, 2024

Scientific Computing and Data Management6 citations

TL;DR

AFLOW uses Monte Carlo Tree Search over code-represented, LLM-invoking workflows to automatically discover agentic workflows, outperforming manual and automated baselines across six benchmarks and enabling smaller models to achieve strong cost-performance. It achieves 5.7% average improvement over state-of-the-art baselines and significant cost-efficiency gains.

ABSTRACT

Large language models (LLMs) have demonstrated remarkable potential in solving complex tasks across diverse domains, typically by employing agentic workflows that follow detailed instructions and operational sequences. However, constructing these workflows requires significant human effort, limiting scalability and generalizability. Recent research has sought to automate the generation and optimization of these workflows, but existing methods still rely on initial manual setup and fall short of achieving fully automated and effective workflow generation. To address this challenge, we reformulate workflow optimization as a search problem over code-represented workflows, where LLM-invoking nodes are connected by edges. We introduce AFlow, an automated framework that efficiently explores this space using Monte Carlo Tree Search, iteratively refining workflows through code modification, tree-structured experience, and execution feedback. Empirical evaluations across six benchmark datasets demonstrate AFlow's efficacy, yielding a 5.7% average improvement over state-of-the-art baselines. Furthermore, AFlow enables smaller models to outperform GPT-4o on specific tasks at 4.55% of its inference cost in dollars. The code is available at https://github.com/FoundationAgents/AFlow.

Motivation & Objective

• Motivate scalable automation of agentic workflows to reduce manual design effort.
• Formulate workflow optimization as a search over code-represented LLM-invoking graphs.
• Develop AFLOW, an MCTS-based framework with operators to efficiently explore the space.
• Demonstrate AFLOW’s effectiveness across multiple benchmarks and cost-efficiency implications.

Proposed method

• Represent workflows as nodes (LLM invocations) connected by code-based edges.
• Use Monte Carlo Tree Search with a soft mixed-probability node selection and LLM-driven expansion.
• Introduce operators (e.g., Generate, Review & Revise, Ensemble) to build reusable workflow patterns.
• Fix key parameters (model, temperature, format) to focus search on edges and prompts.
• Evaluate generated workflows via execution feedback and backpropagate experience to guide search.
• Demonstrate that AFLOW improves over baselines by 5.7% on average and enables smaller models to beat larger ones at lower cost.

Experimental results

Research questions

• RQ1 Can AFLOW automatically discover high-performing agentic workflows across diverse domains without manual intervention?
• RQ2 How does AFLOW’s MCTS-based search compare to existing automated workflow optimization methods in terms of performance and efficiency?
• RQ3 What cost-performance trade-offs arise when using AFLOW with different execution LLMs?
• RQ4 To what extent are the discovered workflows transferrable across different language models?
• RQ5 What is the impact of operators on search efficiency and final workflow quality?

Key findings

• AFLOW yields a 5.7% average improvement over state-of-the-art baselines across six benchmarks.
• AFLOW surpasses existing automated approaches by 19.5%.
• On average, AFLOW workflows achieve 80.3% performance across QA, code, and math domains.
• Workflows discovered by AFLOW enable smaller LLMs to outperform larger models at substantially lower cost (Pareto-front results).
• AFLOW demonstrates model-agnostic transferability of discovered workflows across different executors.
• Ablation shows operators improve search efficiency, though AFLOW can perform well even without predefined operators.