[Paper Review] Rolling Horizon Coevolutionary Planning for Two-Player Video Games

Jialin Liu, Diego Pérez-Liébana+1 more|arXiv (Cornell University)|Jul 6, 2016

Artificial Intelligence in Games3 citations

TL;DR

This paper introduces the Rolling Horizon Coevolutionary Algorithm (RHCA), a novel decision-making approach for two-player real-time video games that extends rolling horizon evolutionary planning to co-evolve action sequences for both players. RHCA outperforms multiple general game AI algorithms, including MCTS and heuristic-based controllers, across multiple variants of a custom two-player space battle game, demonstrating strong performance even without prior training.

ABSTRACT

This paper describes a new algorithm for decision making in two-player real-time video games. As with Monte Carlo Tree Search, the algorithm can be used without heuristics and has been developed for use in general video game AI. The approach is to extend recent work on rolling horizon evolutionary planning, which has been shown to work well for single-player games, to two (or in principle many) player games. To select an action the algorithm co-evolves two (or in the general case N) populations, one for each player, where each individual is a sequence of actions for the respective player. The fitness of each individual is evaluated by playing it against a selection of action-sequences from the opposing population. When choosing an action to take in the game, the first action is chosen from the fittest member of the population for that player. The new algorithm is compared with a number of general video game AI algorithms on three variations of a two-player space battle game, with promising results.

Motivation & Objective

• To develop a general-purpose decision-making algorithm for two-player real-time video games that does not require prior training.
• To extend rolling horizon evolutionary algorithms (RHEA), successful in single-player games, to multi-player settings.
• To evaluate the performance of the proposed algorithm against a range of general game AI methods in a controlled, customizable two-player battle environment.
• To assess the impact of macro-action size (number of actions per sequence) on algorithmic performance.
• To explore the potential of coevolutionary planning as a viable alternative to Monte Carlo Tree Search (MCTS) in real-time games.

Proposed method

• RHCA co-evolves two separate populations of action sequences, one per player, using a rolling horizon framework where each individual represents a sequence of actions for a fixed planning horizon.
• Fitness of each action sequence is evaluated through repeated roll-outs against a sample of opponents' action sequences from the opposing population.
• The first action of the fittest individual in each player's population is selected for execution in the current game state.
• The algorithm uses a fixed simulation depth (planning horizon), and the population is re-evaluated and re-optimized at each game tick, maintaining a rolling window of future actions.
• The method supports stochastic games by averaging results over multiple random initializations and game runs.
• The approach is contrasted with MCTS and other open-loop controllers such as OneStep, OLMCTS, and random or rotated baseline agents.

Experimental results

Research questions

• RQ1 Can rolling horizon evolutionary planning be successfully extended from single-player to two-player real-time games?
• RQ2 How does the performance of RHCA compare to established general game AI algorithms like MCTS and heuristic-based controllers in two-player settings?
• RQ3 Does the size of macro-actions (number of actions per sequence) significantly affect the performance of RHCA and RHGA?
• RQ4 Can coevolutionary planning achieve strong performance without prior training, making it suitable for rapid game design feedback?
• RQ5 Is RHCA a viable and competitive alternative to MCTS in real-time two-player games?

Key findings

• RHCA achieved the highest average win rate of 76.1% across all tested games, outperforming all other controllers in the round-robin tournament.
• In the simple battle game without weapons (G₁), RHCA won 93.5% of games against the random controller (RND), 100% against the rotated controller (ROT), and 87.0% against the OneStep controller.
• RHCA significantly outperformed OLMCTS, which had a 68.3% average win rate, and RHGA, which had a 70.1% average win rate.
• Performance of both RHCA and RHGA degraded as the number of actions per macro-action increased, indicating that shorter planning horizons yield better results.
• OneStep performed poorly overall, winning only 51.0% against RND and 0.0% against ROT, except in the specific case of beating ROT in G₁.
• OLMCTS showed moderate performance, with a 68.3% average win rate, but no clear advantage over RHCA or RHGA in any game variant.