[Paper Review] CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society

Guohao Li, Hasan Abed Al Kader Hammoud+3 more|arXiv (Cornell University)|Mar 31, 2023

Speech and dialogue systemsComputer Science93 citations

TL;DR

The paper introduces CAMEL, a cooperative role-playing framework with inception prompting to enable autonomous multi-agent collaboration, generates large-scale conversational datasets, and shows CAMEL-based solutions outperform single-shot baselines in multiple evaluations.

ABSTRACT

The rapid advancement of chat-based language models has led to remarkable progress in complex task-solving. However, their success heavily relies on human input to guide the conversation, which can be challenging and time-consuming. This paper explores the potential of building scalable techniques to facilitate autonomous cooperation among communicative agents, and provides insight into their "cognitive" processes. To address the challenges of achieving autonomous cooperation, we propose a novel communicative agent framework named role-playing. Our approach involves using inception prompting to guide chat agents toward task completion while maintaining consistency with human intentions. We showcase how role-playing can be used to generate conversational data for studying the behaviors and capabilities of a society of agents, providing a valuable resource for investigating conversational language models. In particular, we conduct comprehensive studies on instruction-following cooperation in multi-agent settings. Our contributions include introducing a novel communicative agent framework, offering a scalable approach for studying the cooperative behaviors and capabilities of multi-agent systems, and open-sourcing our library to support research on communicative agents and beyond: https://github.com/camel-ai/camel.

Motivation & Objective

• Develop a scalable framework for autonomous cooperation among communicative agents to complete complex tasks with minimal human input.
• Study the behaviors and capabilities of a society of agents through role-playing and data generation.
• Provide datasets and open-source tools to enable research in cooperative AI, alignment, and multi-agent systems.
• Investigate the emergence of capabilities in LLMs by fine-tuning models on dataset streams generated by the framework.

Proposed method

• Introduce the role-playing framework where AI assistants and users are assigned distinct roles and collaborate to complete a specified task.
• Use Inception Prompting to generate task specifications and role prompts at the outset, followed by autonomous AI–AI conversations.
• Incorporate a critic-in-the-loop to guide proposals and simulate tree-search-like decision making.
• Generate large-scale conversational datasets (AI Society, Code) and single-turn QA datasets (Math, Science) for analysis and alignment studies.
• Evaluate agent performance against single-shot baselines using human and GPT-4 evaluations.
• Progressively fine-tune LLaMA-7B on the generated datasets to study knowledge emergence across domains.

Experimental results

Research questions

• RQ1 Can autonomous cooperative agents complete complex tasks with minimal human input using a role-playing framework?
• RQ2 What challenges (e.g., role flipping, repetition, flake replies, infinite loops) arise in multi-agent cooperation and how can they be mitigated?
• RQ3 Do datasets generated by CAMEL enable emergence of domain knowledge in subsequent model fine-tuning?
• RQ4 Do cooperative multi-agent solutions outperform single-shot LLM baselines in instruction-following tasks?
• RQ5 What are the ethical and alignment considerations of autonomous communicative agents in a society setting?

Key findings

• CAMEL-generated solutions outperform gpt-3.5-turbo single-shot solutions in both human and GPT-4 evaluations for AI Society tasks.
• The framework enables scalable data generation (AI Society, Code) and multi-domain emergence (Math, Science) when used to fine-tune LLaMA-7B.
• Human evaluators and GPT-4 evaluations largely agree on CAMEL’s superiority over single-shot baselines.
• Diverse datasets facilitate knowledge emergence and improved performance across domains during progressive fine-tuning.
• The approach provides a public library with modular agents, prompts, and data explorers to support cooperative AI research.