[Paper Review] Lemur: Harmonizing Natural Language and Code for Language Agents

Yiheng Xu, Hongjin Su+14 more|arXiv (Cornell University)|Oct 10, 2023

Topic Modeling6 citations

TL;DR

Lemur and Lemur-Chat are open-source language models pre-trained on a code-intensive corpus and fine-tuned for instruction following, achieving state-of-the-art balanced performance in both natural language and code generation. They outperform existing open-source models across 13 agent benchmarks by harmonizing language and code capabilities, significantly narrowing the gap with proprietary models in agent reasoning, tool use, and environment grounding.

ABSTRACT

We introduce Lemur and Lemur-Chat, openly accessible language models optimized for both natural language and coding capabilities to serve as the backbone of versatile language agents. The evolution from language chat models to functional language agents demands that models not only master human interaction, reasoning, and planning but also ensure grounding in the relevant environments. This calls for a harmonious blend of language and coding capabilities in the models. Lemur and Lemur-Chat are proposed to address this necessity, demonstrating balanced proficiencies in both domains, unlike existing open-source models that tend to specialize in either. Through meticulous pre-training using a code-intensive corpus and instruction fine-tuning on text and code data, our models achieve state-of-the-art averaged performance across diverse text and coding benchmarks among open-source models. Comprehensive experiments demonstrate Lemur's superiority over existing open-source models and its proficiency across various agent tasks involving human communication, tool usage, and interaction under fully- and partially- observable environments. The harmonization between natural and programming languages enables Lemur-Chat to significantly narrow the gap with proprietary models on agent abilities, providing key insights into developing advanced open-source agents adept at reasoning, planning, and operating seamlessly across environments. https://github.com/OpenLemur/Lemur

Motivation & Objective

• To address the imbalance in open-source LLMs that specialize in either natural language or code by creating a model with harmonized capabilities.
• To enable language agents to reason, plan, and act effectively in complex, partially observable environments through integrated language and code proficiency.
• To close the performance gap between open-source and proprietary models in agent tasks involving tool use, environment feedback, and multi-step planning.
• To develop a foundation model that excels in both text and code benchmarks while supporting interactive, grounded agent behaviors.
• To provide a publicly available, versatile language model that supports real-world agent applications in web interaction, robotics, and data science.

Proposed method

• Pre-training on a 90-billion-token, code-centric corpus with a 10:1 text-to-code ratio derived from The Stack, enhancing code generation while preserving natural language quality.
• Instruction fine-tuning on ~300K diverse examples combining text and code tasks to build an instruction-following model, resulting in Lemur-Chat.
• Designing a comprehensive agent evaluation suite covering reasoning, code generation, tool use, and environment interaction in fully- and partially-observable settings.
• Evaluating models on 8 standard benchmarks (e.g., HumanEval, GSM8K, MMLU, Spider, DS-1000) and 13 agent-specific benchmarks including WebArena, ALFWorld, and InterCode.
• Using zero-shot and few-shot prompting protocols consistent with official benchmarks to ensure fair and reproducible comparisons.
• Integrating environment feedback and human feedback into agent evaluation to assess robustness and adaptability in real-world scenarios.

Experimental results

Research questions

• RQ1 Can a single open-source language model achieve state-of-the-art performance in both natural language and code generation benchmarks?
• RQ2 How does harmonized language and code capability improve agent performance in complex, interactive environments?
• RQ3 To what extent can open-source models match proprietary models like GPT-4 in agent reasoning, tool use, and environment grounding?
• RQ4 What is the impact of code-intensive pre-training on balancing text and code abilities in large language models?
• RQ5 How do models with balanced capabilities handle partially observable environments requiring iterative information gathering and plan adaptation?

Key findings

• Lemur-Chat outperforms all other open-source models on 12 out of 13 agent benchmarks, demonstrating superior agent capabilities in real-world scenarios.
• The model achieves state-of-the-art averaged performance across 8 standard text and code benchmarks among open-source models, confirming balanced proficiency.
• In agent tasks involving tool use, such as web browsing and OS interaction, Lemur-Chat significantly outperforms models like Llama-2-70B-Chat, especially in complex, high-action-space environments.
• The integration of natural language and code abilities enables Lemur-Chat to generate valid, executable actions and recover from errors through feedback, outperforming models with weaker coding skills.
• Lemur-Chat closes the performance gap with proprietary models like GPT-4 in agent abilities, particularly in reasoning, planning, and environment grounding.
• The model shows strong generalization in multi-step tasks such as CTF and robotics reasoning, where modular planning and error backtracking are essential.