[Paper Review] TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play

Fanqi Lin, Shi‐Yu Huang+3 more|arXiv (Cornell University)|Feb 15, 2023

Sports Analytics and PerformanceEconomics, Econometrics and Finance3 citations

TL;DR

TiZero introduces a self-evolving multi-agent reinforcement learning system that masters the full 11 vs. 11 Google Research Football environment from scratch, using adaptive curriculum learning, a novel challenge-and-generalize self-play strategy, and joint policy optimization. It achieves over 30% higher win rates than prior methods, demonstrating superior coordination and strategic planning in complex multi-agent settings.

ABSTRACT

Multi-agent football poses an unsolved challenge in AI research. Existing work has focused on tackling simplified scenarios of the game, or else leveraging expert demonstrations. In this paper, we develop a multi-agent system to play the full 11 vs. 11 game mode, without demonstrations. This game mode contains aspects that present major challenges to modern reinforcement learning algorithms; multi-agent coordination, long-term planning, and non-transitivity. To address these challenges, we present TiZero; a self-evolving, multi-agent system that learns from scratch. TiZero introduces several innovations, including adaptive curriculum learning, a novel self-play strategy, and an objective that optimizes the policies of multiple agents jointly. Experimentally, it outperforms previous systems by a large margin on the Google Research Football environment, increasing win rates by over 30%. To demonstrate the generality of TiZero's innovations, they are assessed on several environments beyond football; Overcooked, Multi-agent Particle-Environment, Tic-Tac-Toe and Connect-Four.

Motivation & Objective

• Address the challenge of training strong multi-agent policies in the complex 11 vs. 11 Google Research Football environment without expert demonstrations.
• Overcome key obstacles in multi-agent reinforcement learning: multi-agent coordination, long-term planning, sparse rewards, and non-transitive dynamics.
• Develop a scalable, decentralized training framework that enables 10 outfield players to coordinate effectively in a competitive, stochastic environment.
• Demonstrate the generality of the method beyond football by evaluating on Overcooked, MPE, Tic-Tac-Toe, and Connect-Four.
• Achieve state-of-the-art performance in the 11 vs. 11 mode by combining curriculum learning with a novel self-play strategy that enhances policy diversity and robustness.

Proposed method

• Introduces a joint-policy optimization objective that trains multiple agents simultaneously to improve coordination and shared policy performance.
• Employs adaptive curriculum learning that gradually increases environmental complexity to stabilize training and accelerate policy convergence.
• Designs a challenge-and-generalize self-play strategy that alternates between training against stronger opponents and diverse, generalizable policies to improve robustness.
• Uses a self-play opponent pool that dynamically evolves, ensuring agents face a wide range of strategies and reducing overfitting.
• Applies a centralized training with decentralized execution (CTDE) paradigm, enabling efficient multi-agent credit assignment and policy optimization.
• Leverages a deep neural network architecture with shared encoder heads and individual value heads to balance parameter efficiency and policy specialization.

Figure 1. Screenshot of Google Research Football. TiZero’s agents outperform previous systems by leveraging more coordinated strategies, such as passing more often and creating more assists.

Experimental results

Research questions

• RQ1 Can a multi-agent reinforcement learning system achieve superhuman performance in the 11 vs. 11 Google Research Football environment without expert demonstrations?
• RQ2 How does adaptive curriculum learning improve training stability and sample efficiency in complex multi-agent environments?
• RQ3 To what extent does the challenge-and-generalize self-play strategy enhance policy diversity and robustness compared to standard self-play?
• RQ4 Can the joint-policy optimization objective significantly improve multi-agent coordination and long-term strategic planning?
• RQ5 How generalizable are the proposed methodological innovations across diverse multi-agent environments beyond football?

Key findings

• TiZero achieves a win rate increase of over 30% compared to previous methods in the 11 vs. 11 Google Research Football environment, with a significant improvement in goal difference.
• The system demonstrates superior coordination, such as increased passing frequency and assist creation, compared to prior baselines.
• The challenge-and-generalize self-play strategy achieves a diversity index of 8.11 on Tic-Tac-Toe, outperforming baselines (6.65–7.19), indicating more varied and robust opponent policies.
• TiZero reduces GPU memory consumption by 38% on the 10-player GFootball setting compared to MAPPO (121 GB vs. 196 GB), demonstrating improved scalability.
• In multi-environment evaluations, TiZero outperforms or matches state-of-the-art methods on Overcooked, MPE, Tic-Tac-Toe, and Connect-Four, confirming methodological generality.
• Training curves show that TiZero converges faster and achieves higher final performance than MAPPO, MAT, and QMIX across all benchmarks, with faster wall-clock training times.

Figure 2. TiZero’s network architecture. Six types of information are required as input: the controlled player information, player ID, ball information, teammate information, opponent information and current match information. We use six separate MLPs with two (one for the ”player ID”) fully-connect