[Paper Review] Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization

Zhenghao Peng, Quanyi Li+3 more|arXiv (Cornell University)|Oct 26, 2021

Autonomous Vehicle Technology and Safety58 references28 citations

TL;DR

CoPO trains multi-agent controllers for self-driven particle systems by bi-level coordination that blends individual and neighborhood rewards with learned social preferences, yielding safer, more efficient traffic-like emergent behaviors.

ABSTRACT

Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. In a SDP system, each agent pursues its own goal and constantly changes its cooperative or competitive behaviors with its nearby agents. Manually designing the controllers for such SDP system is time-consuming, while the resulting emergent behaviors are often not realistic nor generalizable. Thus the realistic simulation of SDP systems remains challenging. Reinforcement learning provides an appealing alternative for automating the development of the controller for SDP. However, previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the role of each agent varies to be cooperative or competitive even within one episode. To simulate SDP with MARL, a key challenge is to coordinate agents' behaviors while still maximizing individual objectives. Taking traffic simulation as the testing bed, in this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP. Experiments show that the proposed method can achieve superior performance compared to MARL baselines in various metrics. Noticeably the trained vehicles exhibit complex and diverse social behaviors that improve performance and safety of the population as a whole. Demo video and source code are available at: https://decisionforce.github.io/CoPO/

Motivation & Objective

• Motivate realistic simulation of Self-Driven Particles (SDP) where agents are self-interested and dynamically cooperative/competitive.
• Develop a MARL method that can coordinate self-interested agents in continuous action spaces.
• Incorporate social psychology principles to enable local and global coordination for improved population performance.
• Demonstrate the method on realistic traffic-style SDP environments and analyze emergent social behaviors.

Proposed method

• Model SDP as decentralized partially observable MDPs (Dec-POMDPs) with individual rewards and local interactions.
• Introduce Local Coordination Factor (LCF) phi to encode agents' social value orientation and compute a neighborhood reward within a radius dn.
• Define coordinated reward r_i^C,t = cos(phi) r_i,t + sin(phi) r_i^N,t to mix self and neighbor utilities.
• Train with a two-tier objective: (i) local coordination using coordinated advantage A^C_{Φ,i,t}, (ii) global coordination via bi-level optimization that aligns with a global objective J^G.
• Use meta-gradient-based global coordination to optimize the LCF distribution Phi, enabling automatic search for the best social preference.
• Adopt four shared neural networks (policy, individual value, neighborhood value, global value) under a centralized training/decentralized execution framework.
• Employ Mean Field-like neighborhood inputs stabilized by a Mean Field w/ CF design as the best centralized critic variant for continuous actions.

Experimental results

Research questions

• RQ1 Can CoPO outperform independent PPO and other baselines in continuous-action MARL for SDP-like traffic environments?
• RQ2 Does incorporating local social coordination and learned global coordination improve population-level metrics such as success rate, efficiency, and safety?
• RQ3 What emergent social behaviors arise from CoPO-trained populations in complex traffic scenarios?
• RQ4 Is global coordination via meta-learning necessary for robust generalization across different numbers of agents?

Key findings

• CoPO achieves higher average success rates across five traffic environments compared with IPO, MFPO, and CL baselines.
• In the most challenging Intersection environment, CoPO substantially outperforms baselines, indicating improved cooperation under complex interactions.
• CoPO improves safety, reducing critical failures and yielding safer, more coordinated traffic flow.
• Learned populations exhibit diverse social behaviors (yielding, queuing, reversing) and emergent coordination that mitigates congestion and crashes.
• Ablation studies show central MF-based critics are more stable than simple Concat or CF variants, and global coordination via learned LCF distribution outperforms fixed or random LCF settings.