[Paper Review] Adversarial Attacks on Adversarial Bandits

Yuzhe Ma, Zhijin Zhou|arXiv (Cornell University)|Jan 30, 2023

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper proposes adversarial attacks on no-regret adversarial bandit algorithms by perturbing loss signals to force selection of a target suboptimal arm in $T-o(T)$ rounds with only $o(T)$ cumulative attack cost. The attack is victim-agnostic, requiring only knowledge of the regret rate, and is asymptotically optimal, as proven by a matching lower bound on attack cost.

ABSTRACT

We study a security threat to adversarial multi-armed bandits, in which an attacker perturbs the loss or reward signal to control the behavior of the victim bandit player. We show that the attacker is able to mislead any no-regret adversarial bandit algorithm into selecting a suboptimal target arm in every but sublinear (T-o(T)) number of rounds, while incurring only sublinear (o(T)) cumulative attack cost. This result implies critical security concern in real-world bandit-based systems, e.g., in online recommendation, an attacker might be able to hijack the recommender system and promote a desired product. Our proposed attack algorithms require knowledge of only the regret rate, thus are agnostic to the concrete bandit algorithm employed by the victim player. We also derived a theoretical lower bound on the cumulative attack cost that any victim-agnostic attack algorithm must incur. The lower bound matches the upper bound achieved by our attack, which shows that our attack is asymptotically optimal.

Motivation & Objective

• To investigate the vulnerability of adversarial bandit algorithms to reward (loss) signal poisoning by an attacker.
• To design a victim-agnostic attack that manipulates any no-regret bandit algorithm into selecting a specific suboptimal arm.
• To establish theoretical bounds on the minimal cumulative attack cost required for such manipulation.
• To evaluate the effectiveness of the attack on both standard and robust bandit algorithms like Exp3 and ExpRb.

Proposed method

• The attack perturbs the loss signal at each round to mislead the bandit player into favoring a target arm, without knowledge of the internal algorithm.
• The attack strategy is designed to maintain sublinear regret for the victim while ensuring the target arm is selected in $T-o(T)$ rounds.
• The attack cost is kept at $o(T)$ by carefully controlling the magnitude and timing of perturbations based on the known regret rate $O(T^\alpha)$.
• A theoretical lower bound on the cumulative attack cost is derived, showing that no victim-agnostic attack can achieve $T-o(T)$ target selections with less cost.
• The attack is evaluated empirically on Exp3 and the robust ExpRb algorithm under varying attacker budgets.

(a) $T-N_{T}(a^{\dagger})$ of ( 5 ).

Experimental results

Research questions

• RQ1 Can an attacker manipulate any no-regret adversarial bandit algorithm to select a specific suboptimal arm in most rounds?
• RQ2 What is the minimal cumulative attack cost required to achieve $T-o(T)$ target arm selections across all victim-agnostic attacks?
• RQ3 How does the attack perform against robust bandit algorithms like ExpRb that are designed to resist reward corruption?
• RQ4 Is the proposed attack asymptotically optimal in terms of attack cost?

Key findings

• The attack successfully forces any no-regret adversarial bandit algorithm to select the target arm in $T-o(T)$ rounds while incurring only $o(T)$ cumulative attack cost.
• The attack is asymptotically optimal, as the derived lower bound on attack cost matches the upper bound achieved by the proposed method.
• Empirical results show the attack remains effective against both standard Exp3 and the robust ExpRb algorithm, with attack costs of $1.40\times10^5$, $3.62\times10^5$, and $5.15\times10^5$ for attacker budgets $\Phi = T^{0.5}, T^{0.7}, T^{0.9}$.
• The per-round attack cost is $0.14$, $0.36$, and $0.51$ respectively, increasing with the assumed attacker budget, indicating higher robustness of ExpRb under larger budgets.
• The attack remains effective even when the victim uses a robust algorithm like ExpRb, which assumes a fixed attacker budget, showing that existing defenses may not be sufficient against adaptive, victim-agnostic attacks.

(b) $C_{T}$ of ( 5 ).