[Paper Review] Intent-aware Multi-agent Reinforcement Learning

Siyuan Qi, Song‐Chun Zhu|arXiv (Cornell University)|Mar 6, 2018

Reinforcement Learning in Robotics40 references3 citations

TL;DR

This paper proposes an intent-aware multi-agent reinforcement learning framework that plans in the goal space by modeling other agents' intentions and intrinsic values to maximize expected utility. The method uses linear function approximation for efficient, convergent learning and demonstrates human-like cooperation and superior performance in dynamic, non-episodic multi-agent scenarios with real-world transferability.

ABSTRACT

This paper proposes an intent-aware multi-agent planning framework as well as a learning algorithm. Under this framework, an agent plans in the goal space to maximize the expected utility. The planning process takes the belief of other agents' intents into consideration. Instead of formulating the learning problem as a partially observable Markov decision process (POMDP), we propose a simple but effective linear function approximation of the utility function. It is based on the observation that for humans, other people's intents will pose an influence on our utility for a goal. The proposed framework has several major advantages: i) it is computationally feasible and guaranteed to converge. ii) It can easily integrate existing intent prediction and low-level planning algorithms. iii) It does not suffer from sparse feedbacks in the action space. We experiment our algorithm in a real-world problem that is non-episodic, and the number of agents and goals can vary over time. Our algorithm is trained in a scene in which aerial robots and humans interact, and tested in a novel scene with a different environment. Experimental results show that our algorithm achieves the best performance and human-like behaviors emerge during the dynamic process.

Motivation & Objective

• To develop a computationally feasible, convergent multi-agent reinforcement learning framework that incorporates intent reasoning for improved decision-making.
• To decouple intent prediction from high-level planning to enable integration with existing algorithms and avoid POMDP complexity.
• To enable emergence of human-like social behaviors such as cooperation and competition in dynamic, non-stationary environments.
• To design a learning method robust to sparse feedback and adaptable to varying numbers of agents and goals.

Proposed method

• The framework plans in the goal space by estimating the expected utility of goals based on beliefs about other agents' intents and intrinsic values.
• It employs a linear function approximation of the utility function, defined as U(g, b) = θᵀφ(g, b), where φ is a feature vector combining goal and belief representations.
• The learning algorithm uses Sarsa with ϵ-greedy exploration to update the parameter vector θ based on sparse rewards from capturing agents entering buildings.
• Intent beliefs are updated using observed behaviors and can be computed via Bayesian methods, maximum likelihood, or communication.
• The method decouples low-level planning, intent prediction, and high-level reasoning, enabling modular integration of existing algorithms.
• The framework avoids the computational intractability of POMDPs by separating belief updating from policy learning.

Experimental results

Research questions

• RQ1 How can multi-agent reinforcement learning be made computationally feasible while incorporating intent reasoning?
• RQ2 Can intent-aware planning lead to the emergence of human-like cooperative and competitive behaviors in dynamic environments?
• RQ3 How can existing intent prediction and low-level planning algorithms be effectively integrated into a unified learning framework?
• RQ4 Does planning in the goal space mitigate the problem of sparse feedback in the action space?
• RQ5 Can a trained policy generalize to novel environments with different numbers of agents and goals?

Key findings

• The proposed method achieved a 60% capture rate in monitoring 5 buildings with 3 robot agents, significantly outperforming baselines such as RNN-POMDP (20%) and greedy/random policies.
• The algorithm converged to a global optimum after 300 training iterations, demonstrating stable and efficient learning.
• The agents learned to monitor unmonitored buildings and coordinate to cover different buildings, resulting in natural emergence of cooperation.
• The value landscape learned by the policy parameters θ reflected core task values: monitoring unmonitored buildings was rewarding, while monitoring the same building as others was penalized.
• The method successfully transferred to a novel scene with a different environment, achieving the best performance despite no explicit knowledge of the new layout.
• Reward shaping that credits only successful captures led to better cooperation than team-wide rewards, mirroring human team dynamics.