[Paper Review] The 2017 AIBIRDS Competition

Matthew Stephenson, Jochen Renz+2 more|arXiv (Cornell University)|Mar 14, 2018

Artificial Intelligence in Games21 references4 citations

TL;DR

This paper presents the 2017 AIBIRDS competition, which challenged AI agents to solve unseen physics-based levels in Angry Birds using only visual input, simulating real-world physical reasoning. Despite advances in deep reinforcement learning, no agent matched human-level performance, highlighting the need for integrated AI approaches combining vision, planning, and reasoning under uncertainty.

ABSTRACT

This paper presents an overview of the sixth AIBIRDS competition, held at the 26th International Joint Conference on Artificial Intelligence. This competition tasked participants with developing an intelligent agent which can play the physics-based puzzle game Angry Birds. This game uses a sophisticated physics engine that requires agents to reason and predict the outcome of actions with only limited environmental information. Agents entered into this competition were required to solve a wide assortment of previously unseen levels within a set time limit. The physical reasoning and planning required to solve these levels are very similar to those of many real-world problems. This year's competition featured some of the best agents developed so far and even included several new AI techniques such as deep reinforcement learning. Within this paper we describe the framework, rules, submitted agents and results for this competition. We also provide some background information on related work and other video game AI competitions, as well as discussing some potential ideas for future AIBIRDS competitions and agent improvements.

Motivation & Objective

• To advance AI research in physical reasoning by creating a competition where agents must solve unknown Angry Birds levels using only visual input.
• To evaluate the effectiveness of diverse AI techniques, including deep reinforcement learning, in complex, continuous, physics-based environments.
• To identify limitations in current AI methods for reasoning under uncertainty and with incomplete state information.
• To promote collaboration and transparency by encouraging open-sourcing of agent designs and sharing of technical insights.
• To inspire future research by exposing students and researchers to real-world AI challenges through accessible, game-based platforms.

Proposed method

• Agents were required to interact with a physics engine via a computer vision module that provided object boundaries and positions from game screenshots.
• The competition used a real-time, time-limited format where agents could retry levels multiple times and in any order.
• Scoring was based on the maximum points achieved per level within the time limit, with final rankings determined by total cumulative score.
• Agents had no access to internal game state or physical parameters, mimicking human player perception.
• The framework supported diverse AI techniques, including heuristic search, knowledge representation, and deep reinforcement learning.
• Evaluation was conducted across multiple rounds of elimination, with results analyzed to assess performance and identify weaknesses in agent design.

Experimental results

Research questions

• RQ1 Can modern AI techniques such as deep reinforcement learning effectively solve unseen, complex physics-based puzzle levels in Angry Birds?
• RQ2 To what extent do current agents fail to generalize beyond known or standard levels, and what causes this failure?
• RQ3 How do agents perform when relying solely on visual input without access to internal physics parameters?
• RQ4 What are the key limitations of single-technique approaches in physical reasoning tasks?
• RQ5 How can future competitions better identify and address gaps in AI capabilities for real-world physical interaction?

Key findings

• No agent using deep reinforcement learning alone achieved performance comparable to human players, indicating limitations of current deep RL in physical reasoning.
• Many agents showed overfitting to known level types, suggesting poor generalization to novel or creatively designed levels.
• The competition revealed that even advanced AI techniques struggle with reasoning under uncertainty and incomplete state information.
• Agents relying on traditional planning and heuristic search performed better than pure learning-based approaches, especially in complex scenarios.
• The results underscored the necessity of combining multiple AI techniques—such as vision, planning, and reasoning—rather than relying on a single method.
• There was a strong need for more diverse, challenging, and human-easy/computer-hard levels to better evaluate and improve agent capabilities.