[Paper Review] Fair Algorithms for Multi-Agent Multi-Armed Bandits

Safwan Hossain, Evi Micha+1 more|arXiv (Cornell University)|Jul 13, 2020

Advanced Bandit Algorithms Research33 references16 citations

TL;DR

This paper introduces a multi-agent multi-armed bandit framework where each agent assigns different stochastic rewards to arms, and proposes fair learning algorithms that maximize Nash social welfare (NSW) as a fairness criterion. It designs multi-agent variants of Explore-First, Epsilon-Greedy, and UCB, proving they achieve sublinear regret in terms of lost NSW, ensuring fair long-term distribution of arm pulls across agents with diverse preferences.

ABSTRACT

We propose a multi-agent variant of the classical multi-armed bandit problem, in which there are $N$ agents and $K$ arms, and pulling an arm generates a (possibly different) stochastic reward for each agent. Unlike the classical multi-armed bandit problem, the goal is not to learn the "best arm"; indeed, each agent may perceive a different arm to be the best for her personally. Instead, we seek to learn a fair distribution over the arms. Drawing on a long line of research in economics and computer science, we use the Nash social welfare as our notion of fairness. We design multi-agent variants of three classic multi-armed bandit algorithms and show that they achieve sublinear regret, which is now measured in terms of the lost Nash social welfare.

Motivation & Objective

• To address fairness in multi-agent decision-making where agents have conflicting preferences over arms.
• To model collective policy selection in settings affecting multiple stakeholders, such as corporate decisions or conference review systems.
• To replace utilitarian or egalitarian fairness with Nash social welfare as a balanced fairness criterion.
• To design learning algorithms that converge to a distribution over arms maximizing NSW despite unknown reward distributions.
• To establish regret bounds in terms of lost Nash social welfare, ensuring sublinear performance growth with time horizon T.

Proposed method

• Adapts classical multi-armed bandit algorithms—Explore-First, Epsilon-Greedy, and UCB—for multi-agent settings with agent-specific rewards.
• Defines regret as the cumulative loss in Nash social welfare relative to the optimal distribution over arms.
• Uses empirical mean estimates of agent-specific rewards to guide arm selection, with exploration balancing to ensure convergence.
• For UCB, introduces an upper confidence bound that combines estimated NSW and exploration bonus to balance exploitation and exploration.
• Employs a logarithmic transformation of NSW in the UCB variant to enable efficient optimization, though regret bounds are derived for the original NSW function.
• Proves that the proposed algorithms achieve sublinear regret in T, with the regret growing slower than T, even under agent-wise reward heterogeneity.

Experimental results

Research questions

• RQ1 Can we design learning algorithms for multi-agent multi-armed bandits that achieve fairness when agents have conflicting preferences over arms?
• RQ2 Does maximizing Nash social welfare yield a fair distribution of arm pulls that respects minority preferences without succumbing to the tyranny of the majority?
• RQ3 Can classical bandit algorithms be adapted to maintain sublinear regret when regret is measured in terms of lost Nash social welfare?
• RQ4 Is the regret of the proposed algorithms bounded in a way that scales gracefully with the number of agents and arms?
• RQ5 What are the theoretical limits of regret in this multi-agent fairness setting, and can tighter lower bounds be derived?

Key findings

• The proposed multi-agent variants of Explore-First, Epsilon-Greedy, and UCB all achieve sublinear regret in the time horizon T, measured as the cumulative loss in Nash social welfare.
• The regret bound for the Explore-First and Epsilon-Greedy variants is O(√(KT log T)), matching known bounds in the classical MAB setting.
• For the UCB variant, the regret is bounded by O(√(KT log T)), though the computational complexity of the optimization step remains open for general N.
• A lower bound of Ω(√(KT)) is established for any algorithm in the multi-agent setting, showing that the upper bounds are nearly tight.
• The use of Nash social welfare ensures proportionally fair outcomes, such as assigning 40% and 60% probability to arms in a minority-majority preference scenario.
• The paper identifies that deriving an instance-dependent logarithmic regret bound for UCB in this setting remains an open problem.