[论文解读] CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society

Guohao Li, Hasan Abed Al Kader Hammoud+3 more|arXiv (Cornell University)|Mar 31, 2023

Speech and dialogue systems被引用 93

一句话总结

本文介绍 CAMEL，这是一个具有 inception prompting 的协作角色扮演框架，使自主多智能体协作成为可能，生成大规模对话数据集，并显示基于 CAMEL 的解决方案在多项评估中优于单次基线。

ABSTRACT

The rapid advancement of chat-based language models has led to remarkable progress in complex task-solving. However, their success heavily relies on human input to guide the conversation, which can be challenging and time-consuming. This paper explores the potential of building scalable techniques to facilitate autonomous cooperation among communicative agents, and provides insight into their "cognitive" processes. To address the challenges of achieving autonomous cooperation, we propose a novel communicative agent framework named role-playing. Our approach involves using inception prompting to guide chat agents toward task completion while maintaining consistency with human intentions. We showcase how role-playing can be used to generate conversational data for studying the behaviors and capabilities of a society of agents, providing a valuable resource for investigating conversational language models. In particular, we conduct comprehensive studies on instruction-following cooperation in multi-agent settings. Our contributions include introducing a novel communicative agent framework, offering a scalable approach for studying the cooperative behaviors and capabilities of multi-agent systems, and open-sourcing our library to support research on communicative agents and beyond: https://github.com/camel-ai/camel.

研究动机与目标

• Develop a scalable framework for autonomous cooperation among communicative agents to complete complex tasks with minimal human input.
• Study the behaviors and capabilities of a society of agents through role-playing and data generation.
• Provide datasets and open-source tools to enable research in cooperative AI, alignment, and multi-agent systems.
• Investigate the emergence of capabilities in LLMs by fine-tuning models on dataset streams generated by the framework.

提出的方法

• Introduce the role-playing framework where AI assistants and users are assigned distinct roles and collaborate to complete a specified task.
• Use Inception Prompting to generate task specifications and role prompts at the outset, followed by autonomous AI–AI conversations.
• Incorporate a critic-in-the-loop to guide proposals and simulate tree-search-like decision making.
• Generate large-scale conversational datasets (AI Society, Code) and single-turn QA datasets (Math, Science) for analysis and alignment studies.
• Evaluate agent performance against single-shot baselines using human and GPT-4 evaluations.
• Progressively fine-tune LLaMA-7B on the generated datasets to study knowledge emergence across domains.

实验结果

研究问题

• RQ1 Can autonomous cooperative agents complete complex tasks with minimal human input using a role-playing framework?
• RQ2 What challenges (e.g., role flipping, repetition, flake replies, infinite loops) arise in multi-agent cooperation and how can they be mitigated?
• RQ3 Do datasets generated by CAMEL enable emergence of domain knowledge in subsequent model fine-tuning?
• RQ4 Do cooperative multi-agent solutions outperform single-shot LLM baselines in instruction-following tasks?
• RQ5 What are the ethical and alignment considerations of autonomous communicative agents in a society setting?

主要发现

数据集	评估类型	和局	gpt-3.5-turbo 胜利	CAMEL Agents 胜利
AI Society	人工评估	13.3%	10.4%	76.3%
AI Society	GPT4 评估	4.0%	23.0%	73.0%
Code	GPT4 评估	0.0%	24.0%	76.0%

• CAMEL-generated solutions outperform gpt-3.5-turbo single-shot solutions in both human and GPT-4 evaluations for AI Society tasks.
• The framework enables scalable data generation (AI Society, Code) and multi-domain emergence (Math, Science) when used to fine-tune LLaMA-7B.
• Human evaluators and GPT-4 evaluations largely agree on CAMEL’s superiority over single-shot baselines.
• Diverse datasets facilitate knowledge emergence and improved performance across domains during progressive fine-tuning.
• The approach provides a public library with modular agents, prompts, and data explorers to support cooperative AI research.