[Paper Review] Collaborative Top Distribution Identifications with Limited Interaction

Nikolai Karpov, Qin Zhang+1 more|arXiv (Cornell University)|Apr 20, 2020

Advanced Bandit Algorithms Research59 references4 citations

TL;DR

This paper presents an optimal collaborative algorithm for identifying the top-$m$ arms (distributions with the highest means) using limited interaction among $K$ agents, achieving near-optimal running time $\tilde{O}(H^{\langle m\rangle}/K)$ with $O(\log(1/\Delta^{\langle m\rangle}))$ rounds. It establishes tight time-round tradeoffs and reveals fundamental complexity gaps between top-$m$ and top-1 identification, as well as between fixed-time and fixed-confidence settings.

ABSTRACT

We consider the following problem in this paper: given a set of $n$ distributions, find the top-$m$ ones with the largest means. This problem is also called {\em top-$m$ arm identifications} in the literature of reinforcement learning, and has numerous applications. We study the problem in the collaborative learning model where we have multiple agents who can draw samples from the $n$ distributions in parallel. Our goal is to characterize the tradeoffs between the running time of learning process and the number of rounds of interaction between agents, which is very expensive in various scenarios. We give optimal time-round tradeoffs, as well as demonstrate complexity separations between top-$1$ arm identification and top-$m$ arm identifications for general $m$ and between fixed-time and fixed-confidence variants. As a byproduct, we also give an algorithm for selecting the distribution with the $m$-th largest mean in the collaborative learning model.

Motivation & Objective

• To address the challenge of identifying the top-$m$ distributions with the largest means in a collaborative multi-agent setting with minimal communication rounds.
• To characterize the tradeoff between running time and number of communication rounds in collaborative learning under limited interaction.
• To establish separations in complexity between top-$m$ and top-1 arm identification, and between fixed-time and fixed-confidence variants.
• To design an algorithm that achieves near-optimal running time while minimizing rounds, even when communication is expensive.
• To provide a method for identifying the $m$-th largest mean distribution in the collaborative model.

Proposed method

• The algorithm uses a recursive elimination framework where agents iteratively refine a candidate set of arms based on empirical means and confidence intervals.
• In each round $r$, agents sample arms in $I_r$ with $T_r = O(\log(1/\delta)/(K\epsilon_r^2))$ pulls per agent, where $\epsilon_r$ decreases geometrically.
• Agents communicate at the end of each round to update the active set $I_{r+1}$, discarding arms that are unlikely to be in the top-$m$ based on confidence bounds.
• The algorithm maintains three invariants: $m_r = |\text{Top}_m \cap I_r|$, $\text{Acc}_r \subseteq \text{Top}_m$, and $\text{Rej}_r \subseteq I \setminus \text{Top}_m$.
• It uses concentration inequalities (Hoeffding's bound) to ensure high-probability correctness, with error bounds controlled via union bounds over rounds and arms.
• The stopping condition is triggered when $\epsilon_r \leq \Delta^{\langle m\rangle}_i / 4$ for all arms, ensuring all top-$m$ arms are correctly identified.

Experimental results

Research questions

• RQ1 What is the optimal tradeoff between running time and number of communication rounds for collaborative top-$m$ arm identification?
• RQ2 How does the complexity of top-$m$ identification differ from top-1 identification in the multi-agent setting?
• RQ3 Can fixed-time and fixed-confidence variants of top-$m$ identification be separated in terms of round complexity?
• RQ4 Is there a way to identify the $m$-th largest mean arm efficiently in a collaborative setting with limited interaction?
• RQ5 What is the minimal number of rounds required to achieve near-optimal speedup in collaborative learning?

Key findings

• The proposed algorithm achieves a running time of $O\left(\frac{H^{\langle m\rangle}}{K}\log\left(\frac{n}{\delta}\log H^{\langle m\rangle}\right)\right)$ with $O(\log(1/\Delta^{\langle m\rangle}))$ rounds, matching the lower bound up to logarithmic factors.
• The algorithm correctly identifies the top-$m$ arms with probability at least $1-\delta$ under the event $\mathcal{E}_3$, which holds with high probability.
• A complexity separation exists between top-$m$ and top-1 arm identification: the round complexity for top-$m$ is strictly higher than for top-1 when $m > 1$.
• The fixed-confidence variant of top-$m$ identification requires $\Omega(\log(1/\Delta^{\langle m\rangle}))$ rounds, matching the upper bound, showing tightness.
• The algorithm also enables identification of the $m$-th largest mean arm by maintaining a refined active set through rounds.
• The time complexity is optimal up to logarithmic factors, and the round complexity is tight for achieving near-full speedup in collaborative learning.