[Paper Review] Asynchronous Upper Confidence Bound Algorithms for Federated Linear Bandits

Chuanhao Li, Hongning Wang|arXiv (Cornell University)|Oct 4, 2021

Advanced Bandit Algorithms Research2 references4 citations

TL;DR

This paper proposes asynchronous Upper Confidence Bound (UCB) algorithms for federated linear bandits, enabling efficient, communication-optimized learning across heterogeneous clients without global synchronization. By triggering updates only when model changes become relevant, the method reduces communication costs while maintaining low regret, with empirical results showing improved performance through selective data sharing from active users.

ABSTRACT

Linear contextual bandit is a popular online learning problem. It has been mostly studied in centralized learning settings. With the surging demand of large-scale decentralized model learning, e.g., federated learning, how to retain regret minimization while reducing communication cost becomes an open challenge. In this paper, we study linear contextual bandit in a federated learning setting. We propose a general framework with asynchronous model update and communication for a collection of homogeneous clients and heterogeneous clients, respectively. Rigorous theoretical analysis is provided about the regret and communication cost under this distributed learning framework; and extensive empirical evaluations demonstrate the effectiveness of our solution.

Motivation & Objective

• To address the challenge of minimizing regret in federated linear bandits while reducing communication overhead in decentralized, large-scale learning settings.
• To design a robust, asynchronous communication framework that avoids global synchronization, improving resilience to client delays and unavailability.
• To support both homogeneous and heterogeneous clients—where clients may have distinct reward functions and data distributions—within a unified learning framework.
• To theoretically analyze and empirically validate the regret and communication cost of the proposed algorithms under varying client activity and data heterogeneity.

Proposed method

• Proposes an event-triggered, asynchronous communication mechanism where clients update the server only when their local model becomes sufficiently outdated relative to the global model.
• Introduces a threshold-based condition (parameterized by γ) to determine when a client's local update is worth transmitting, based on the potential regret reduction from communication.
• Develops two algorithm variants: one for homogeneous clients sharing a common reward function, and another for heterogeneous clients with shared global parameters in a multi-task learning setup.
• Employs a linear contextual bandit framework with upper confidence bound (UCB) exploration, where each client selects actions based on its local estimate of the expected reward.
• Uses PCA to visualize user embeddings and analyze the impact of data heterogeneity on performance, particularly the trade-off between bias and variance from aggregated data.
• Theoretical analysis derives upper bounds on cumulative regret and communication cost, proving the method's efficiency and convergence under both IID and non-IID data assumptions.

Experimental results

Research questions

• RQ1 How can we reduce communication cost in federated linear bandits without relying on global synchronization?
• RQ2 What conditions ensure that a client’s local model update contributes meaningfully to regret reduction, justifying communication?
• RQ3 How does data heterogeneity—especially from less active clients—affect the performance of the global model in asynchronous federated bandit learning?
• RQ4 Can selective data sharing from highly active users improve overall performance despite reduced communication?
• RQ5 What is the trade-off between bias (from non-IID data) and variance (from limited data) in the context of asynchronous federated bandit learning?

Key findings

• When the communication threshold γ is high (e.g., ∞), only the most active users contribute to global updates, resulting in a cumulative reward of 1.6891 with only 14,230 communications.
• As γ decreases and more clients participate in communication (e.g., γ = 3), the cumulative reward increases to 1.8348 at 14,230 communications, indicating performance gains from selective data sharing.
• However, when communication increases further (e.g., γ = 3, C_T = 54,006), the cumulative reward drops significantly for less active groups (e.g., from 22 to 11), indicating that excessive data aggregation harms performance due to high heterogeneity.
• The visualization of user embeddings shows that highly active users (80–100) are centrally located in the feature space, while less active users are scattered along orthogonal directions, explaining why their data degrades model performance when aggregated.
• The results confirm that data from active users improves performance across groups, but aggregating data from heterogeneous, less active users increases bias and reduces overall effectiveness.
• The asynchronous framework achieves lower communication cost and better regret performance than synchronous counterparts, especially in environments with variable client availability and data sparsity.