[Paper Review] Attacking c-MARL More Effectively: A Data Driven Approach

Nhan H. Pham, Lam M. Nguyen+4 more|arXiv (Cornell University)|Feb 7, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes c-MBA, the first model-based adversarial attack framework for cooperative multi-agent reinforcement learning (c-MARL), which outperforms existing model-free baselines by crafting stronger state perturbations using a two-step optimization process with proximal gradient methods. It introduces a data-driven failure state definition and an adaptive victim selection strategy, achieving up to 8–9× reward reduction in c-MARL environments with minimal environment knowledge.

ABSTRACT

In recent years, a proliferation of methods were developed for cooperative multi-agent reinforcement learning (c-MARL). However, the robustness of c-MARL agents against adversarial attacks has been rarely explored. In this paper, we propose to evaluate the robustness of c-MARL agents via a model-based approach, named c-MBA. Our proposed formulation can craft much stronger adversarial state perturbations of c-MARL agents to lower total team rewards than existing model-free approaches. In addition, we propose the first victim-agent selection strategy and the first data-driven approach to define targeted failure states where each of them allows us to develop even stronger adversarial attack without the expert knowledge to the underlying environment. Our numerical experiments on two representative MARL benchmarks illustrate the advantage of our approach over other baselines: our model-based attack consistently outperforms other baselines in all tested environments.

Motivation & Objective

• To address the lack of robustness evaluation in cooperative multi-agent reinforcement learning (c-MARL) against adversarial attacks.
• To develop a model-based attack framework that generates stronger adversarial perturbations than existing model-free approaches.
• To reduce dependency on expert knowledge of the environment by introducing a data-driven method to define failure states.
• To enhance attack effectiveness through an adaptive victim agent selection strategy that targets the most vulnerable agents.
• To evaluate the attack’s performance across diverse c-MARL benchmarks under varying perturbation norms and dynamics model accuracies.

Proposed method

• Formulates the adversarial attack as a two-step optimization problem to generate state perturbations that minimize team reward.
• Uses proximal gradient methods to efficiently solve for adversarial perturbations under ℓ∞ and ℓ1 norm constraints.
• Introduces a data-driven approach to define targeted failure states using pre-collected environment data, eliminating the need for expert-defined states.
• Proposes an adaptive victim selection strategy that identifies the most vulnerable agents to maximize attack impact within a given budget.
• Employs a learned dynamics model to simulate state transitions and guide perturbation generation, improving attack precision.
• Trains dynamics models on collected rollouts and evaluates their impact on attack performance under varying accuracy levels.

Figure 1: We illustrate the proposed model-based attack is powerful while other model-free attacks failed on attacking Agent $0$ in Ant (4x2) environment. The episode ends after 440 time steps under our c-MBA (the agent flips), demonstrating the effectiveness of our algorithm.

Experimental results

Research questions

• RQ1 Can a model-based attack framework outperform existing model-free baselines in attacking c-MARL agents?
• RQ2 How effective is a data-driven approach to defining failure states compared to expert-defined states in c-MARL attacks?
• RQ3 Does an adaptive victim selection strategy significantly improve the strength of adversarial attacks in multi-agent settings?
• RQ4 How does the accuracy of the dynamics model affect the performance of the model-based attack?
• RQ5 Can the proposed attack achieve substantial reward reduction with minimal perturbation budgets in complex c-MARL environments?

Key findings

• c-MBA consistently outperforms all model-free baselines across MA-MuJoCo and MPE benchmarks, reducing team rewards by up to 8–9×.
• The data-driven failure state definition achieves performance comparable to or better than expert-defined failure states in most environments.
• The adaptive victim selection strategy improves attack effectiveness by up to 80% in reward reduction compared to fixed victim selection.
• c-MBA with ℓ1 norm constraints reduced team rewards by 156% more than Lin et al. (2020) + iFGSM in HalfCheetah(6x1).
• Even a dynamics model trained on only 200k samples for 1 epoch achieved comparable attack performance to models trained on 1M samples, demonstrating robustness to model accuracy.
• In environments with more agents, such as HalfCheetah(6x1) and Ant(4x2), c-MBA showed greater advantage under smaller perturbation budgets.

Figure 3: c-MBA vs baselines when attacking one agent in HalfCheetah(6x1) environment - Exp. (I) .