[Paper Review] Decentralized Cooperative Stochastic Bandits

David Martı́nez-Rubio, Varun Kanade+1 more|arXiv (Cornell University)|Oct 10, 2018

Age of Information Optimization54 citations

TL;DR

Proposes a fully decentralized algorithm (DDUCB) for cooperative stochastic multi-armed bandits over networks, using accelerated gossip to estimate global rewards and an UCB strategy that accounts for delay and estimation error; provides regret bounds dependent on the spectral gap.

ABSTRACT

We study a decentralized cooperative stochastic multi-armed bandit problem with $K$ arms on a network of $N$ agents. In our model, the reward distribution of each arm is the same for each agent and rewards are drawn independently across agents and time steps. In each round, each agent chooses an arm to play and subsequently sends a message to her neighbors. The goal is to minimize the overall regret of the entire network. We design a fully decentralized algorithm that uses an accelerated consensus procedure to compute (delayed) estimates of the average of rewards obtained by all the agents for each arm, and then uses an upper confidence bound (UCB) algorithm that accounts for the delay and error of the estimates. We analyze the regret of our algorithm and also provide a lower bound. The regret is bounded by the optimal centralized regret plus a natural and simple term depending on the spectral gap of the communication matrix. Our algorithm is simpler to analyze than those proposed in prior work and it achieves better regret bounds, while requiring less information about the underlying network. It also performs better empirically.

Motivation & Objective

• Study a decentralized stochastic multi-armed bandit problem over a network of N agents each pulling K arms synchronously.
• Develop a fully decentralized algorithm that uses accelerated gossip to estimate global arm-reward statistics.
• Incorporate delayed and noisy estimates into an upper confidence bound (UCB) framework and analyze regret.
• Provide a lower bound and compare asymptotically with centralized and non-communicating baselines.
• Demonstrate improved regret bounds and practical performance with minimal network information requirements.

Proposed method

• Model the problem as N agents on an undirected connected graph with shared arm distributions.
• Use running consensus (gossip) to estimate m_t^k (total rewards for arm k) and n_t^k (total pulls for arm k) across the network, via accelerated Chebyshev-based mixing (polynomial q_C(P)).
• Adapt UCB to delayed/approximate statistics: choose arm by maximizing alpha_i^k/a_i^k + sqrt( (2 eta sigma^2 ln s) / (N a_i^k) ).
• Operate in stages of C communication steps to mix estimates, then update and pull arms in a staged fashion (gamma and beta/beta mixing).
• Provide finite-time regret bounds R(T) with a term depending on the spectral gap |lambda_2| of the communication matrix P and the number of nodes N; prove asymptotic bound and a lower-bound comparison.
• Assume subgaussian rewards with variance proxy sigma^2 and know N and an upper bound on the spectral gap; require lambda_2 (or an upper bound) to set the mixing parameter C.

Experimental results

Research questions

• RQ1 How does decentralization (via gossip) affect regret in stochastic MAB when rewards are shared only through local communications?
• RQ2 Can an accelerated gossip-based estimator enable regret close to centralized UCB with a quantifiable dependence on the graph's spectral properties?
• RQ3 What are the finite-time and asymptotic regret bounds for the proposed DDUCB algorithm in terms of K, N, T, sigma, and lambda_2?
• RQ4 How does the algorithm compare to prior coop-UCB variants in terms of information requirements and performance on general graphs?

Key findings

• The proposed DDUCB algorithm achieves regret equal to the optimal centralized regret plus a term depending on the spectral gap and the number of agents.
• Finite-time regret bound: R(T) < sum_{k: Delta_k>0} (32(1+1/11) sigma^2 ln(TN))/Delta_k + (N(6C+1)+4) sum_k Delta_k, with C depending on lambda_2.
• Asymptotically, R(T) lesssim sum_{k: Delta_k>0} (sigma^2 ln(TN))/Delta_k + (N ln N)/sqrt(ln(1/|lambda_2|)) sum_k Delta_k.
• The lower bound indicates any consistent policy has at least Omega( sum_{k: Delta_k>0} (sigma^2 ln(TN))/Delta_k + (N/K + 1) sum_k Delta_k ), and the DDUCB bound is asymptotically optimal up to a factor depending on K and N.
• DDUCB uses fewer graph-wide assumptions and accelerates communication, improving regret dependence on the spectral gap compared to prior work (e.g., coop-UCB variants).
• Empirically, the method performs better than prior decentralized approaches and is simpler to analyze.