[论文解读] Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization

Zhenghao Peng, Quanyi Li+3 more|arXiv (Cornell University)|Oct 26, 2021

Autonomous Vehicle Technology and Safety参考文献 58被引用 28

一句话总结

CoPO 通过双层协同，将个体与邻域奖励以及学得的社会偏好相结合，训练多智能体控制器以实现自驱体粒子系统的更安全、更高效的交通式涌现行为。

ABSTRACT

Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. In a SDP system, each agent pursues its own goal and constantly changes its cooperative or competitive behaviors with its nearby agents. Manually designing the controllers for such SDP system is time-consuming, while the resulting emergent behaviors are often not realistic nor generalizable. Thus the realistic simulation of SDP systems remains challenging. Reinforcement learning provides an appealing alternative for automating the development of the controller for SDP. However, previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the role of each agent varies to be cooperative or competitive even within one episode. To simulate SDP with MARL, a key challenge is to coordinate agents' behaviors while still maximizing individual objectives. Taking traffic simulation as the testing bed, in this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP. Experiments show that the proposed method can achieve superior performance compared to MARL baselines in various metrics. Noticeably the trained vehicles exhibit complex and diverse social behaviors that improve performance and safety of the population as a whole. Demo video and source code are available at: https://decisionforce.github.io/CoPO/

研究动机与目标

• Motivate realistic simulation of Self-Driven Particles (SDP) where agents are self-interested and dynamically cooperative/competitive.
• Develop a MARL method that can coordinate self-interested agents in continuous action spaces.
• Incorporate social psychology principles to enable local and global coordination for improved population performance.
• Demonstrate the method on realistic traffic-style SDP environments and analyze emergent social behaviors.

提出的方法

• Model SDP as decentralized partially observable MDPs (Dec-POMDPs) with individual rewards and local interactions.
• Introduce Local Coordination Factor (LCF) phi to encode agents' social value orientation and compute a neighborhood reward within a radius dn.
• Define coordinated reward r_i^C,t = cos(phi) r_i,t + sin(phi) r_i^N,t to mix self and neighbor utilities.
• Train with a two-tier objective: (i) local coordination using coordinated advantage A^C_{Φ,i,t}, (ii) global coordination via bi-level optimization that aligns with a global objective J^G.
• Use meta-gradient-based global coordination to optimize the LCF distribution Phi, enabling automatic search for the best social preference.
• Adopt four shared neural networks (policy, individual value, neighborhood value, global value) under a centralized training/decentralized execution framework.
• Employ Mean Field-like neighborhood inputs stabilized by a Mean Field w/ CF design as the best centralized critic variant for continuous actions.

实验结果

研究问题

• RQ1 Can CoPO outperform independent PPO and other baselines in continuous-action MARL for SDP-like traffic environments?
• RQ2 Does incorporating local social coordination and learned global coordination improve population-level metrics such as success rate, efficiency, and safety?
• RQ3 What emergent social behaviors arise from CoPO-trained populations in complex traffic scenarios?
• RQ4 Is global coordination via meta-learning necessary for robust generalization across different numbers of agents?

主要发现

• CoPO achieves higher average success rates across five traffic environments compared with IPO, MFPO, and CL baselines.
• In the most challenging Intersection environment, CoPO substantially outperforms baselines, indicating improved cooperation under complex interactions.
• CoPO improves safety, reducing critical failures and yielding safer, more coordinated traffic flow.
• Learned populations exhibit diverse social behaviors (yielding, queuing, reversing) and emergent coordination that mitigates congestion and crashes.
• Ablation studies show central MF-based critics are more stable than simple Concat or CF variants, and global coordination via learned LCF distribution outperforms fixed or random LCF settings.