[Paper Review] Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence Agents

Stephen A. Goss, Robert J. Steininger+8 more|arXiv (Cornell University)|Jan 27, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

Polycraft World AI Lab (PAL) is an extensible, API-based platform built on the Minecraft mod Polycraft World to evaluate AI agents across diverse tasks. It enables flexible task creation, real-time logging of agent and NPC actions, and supports training and evaluation of agents in complex, open-world environments with customizable dynamics.

ABSTRACT

As artificial intelligence research advances, the platforms used to evaluate AI agents need to adapt and grow to continue to challenge them. We present the Polycraft World AI Lab (PAL), a task simulator with an API based on the Minecraft mod Polycraft World. Our platform is built to allow AI agents with different architectures to easily interact with the Minecraft world, train and be evaluated in multiple tasks. PAL enables the creation of tasks in a flexible manner as well as having the capability to manipulate any aspect of the task during an evaluation. All actions taken by AI agents and external actors (non-player-characters, NPCs) in the open-world environment are logged to streamline evaluation. Here we present two custom tasks on the PAL platform, one focused on multi-step planning and one focused on navigation, and evaluations of agents solving them. In summary, we report a versatile and extensible AI evaluation platform with a low barrier to entry for AI researchers to utilize.

Motivation & Objective

• Address the growing need for adaptable, scalable platforms to evaluate increasingly complex AI agents as research advances.
• Overcome limitations of existing AI evaluation environments by providing extensibility, configurability, and fine-grained control over task dynamics.
• Enable researchers to easily create, modify, and evaluate AI agents across diverse tasks such as multi-step planning and navigation.
• Support interoperability across different AI agent architectures through a standardized API interface.
• Streamline evaluation by logging all agent and NPC actions in real time for reproducible, traceable performance analysis.

Proposed method

• Leverage the Polycraft World mod as a foundation for a persistent, open-world simulation environment with rich, interactive elements.
• Design a modular API that allows AI agents to interact with the environment using discrete actions (e.g., move, craft, interact).
• Implement dynamic task generation by enabling runtime configuration of task parameters, goals, and environmental conditions.
• Log all agent and NPC actions in structured format to support detailed evaluation and analysis of behavior sequences.
• Support integration of non-player characters (NPCs) to simulate complex, multi-agent interactions during evaluation.
• Use the platform to prototype and evaluate two custom tasks: one emphasizing multi-step planning and another focused on navigation.

Experimental results

Research questions

• RQ1 How can an AI evaluation platform be designed to support diverse, customizable tasks with minimal configuration overhead?
• RQ2 To what extent can an open-world environment with dynamic task parameters support the training and evaluation of generalist AI agents?
• RQ3 Can the platform effectively log and analyze complex agent behaviors across multiple interactions and task types?
• RQ4 How does the extensibility of the platform enable rapid prototyping of new evaluation benchmarks?
• RQ5 What are the performance characteristics of different AI agents when evaluated under controlled, configurable task conditions?

Key findings

• PAL enables the creation and execution of complex, customizable tasks with fine-grained control over environmental and goal parameters.
• The platform successfully logs all agent and NPC actions, providing a detailed, traceable record essential for performance evaluation.
• Two custom tasks—multi-step planning and navigation—were implemented and used to evaluate AI agents, demonstrating the platform's versatility.
• The API-based architecture allows seamless integration of diverse AI agent architectures, lowering the barrier to entry for researchers.
• The platform supports real-time manipulation of task conditions during evaluation, enabling dynamic adaptation and robustness testing.
• PAL provides a scalable and extensible foundation for future development of AI evaluation benchmarks in open-world environments.