[Paper Review] Stochastic bandits on a social network: Collaborative learning with local information sharing.

Ravi Kumar Kolla, Krishna Jagannathan+1 more|arXiv (Cornell University)|Feb 29, 2016

Advanced Bandit Algorithms Research11 references3 citations

TL;DR

This paper studies collaborative multi-armed bandit learning on a social network, where agents share rewards with neighbors. It shows that naive extensions of single-agent policies fail due to non-altruistic behavior, but leveraging network structure—especially a star topology—enables the hub to act as an information sink, significantly reducing regret.

ABSTRACT

We consider a collaborative online learning paradigm, wherein a group of agents connected through a social network are engaged in learning a Multi-Armed Bandit problem. Each time an agent takes an action, the corresponding reward is instantaneously observed by the agent, as well as its neighbours in the social network. We perform a regret analysis of various policies in this collaborative learning setting. A key finding of this paper is that appropriate network extensions of widely-studied single agent learning policies do not perform well in terms of regret. In particular, we identify a class of non-altruistic and individually consistent policies, which could suffer a large regret. We also show that the regret performance can be substantially improved by exploiting the network structure. Specifically, we consider a star network, which is a common motif in hierarchical social networks, and show that the hub agent can be used as an information sink, to aid the learning rates of the entire network. We also present numerical experiments to corroborate our analytical results.

Motivation & Objective

• To analyze regret performance in collaborative multi-armed bandit learning where agents share rewards with neighbors via a social network.
• To investigate whether standard single-agent bandit policies generalize well to networked, collaborative settings.
• To identify structural properties of social networks that can enhance learning efficiency and reduce regret.
• To demonstrate how network topology—particularly the star configuration—can be exploited to improve collective learning.

Proposed method

• Formalizing a collaborative learning model where agents in a social network observe rewards not only for their own actions but also for their neighbors’ actions.
• Analyzing regret performance of various policies, including non-altruistic and individually consistent strategies, under networked information sharing.
• Introducing a network extension of bandit policies that accounts for local information sharing, with a focus on star-structured networks.
• Designing a policy where the hub agent in a star network aggregates and disseminates information, functioning as an information sink to accelerate learning.
• Using regret analysis to compare performance across different policy types and network topologies.
• Conducting numerical experiments to validate theoretical findings on regret reduction through structured information sharing.

Experimental results

Research questions

• RQ1 Do standard single-agent bandit policies generalize effectively to collaborative settings with local information sharing?
• RQ2 How does the structure of a social network influence collective regret in collaborative bandit learning?
• RQ3 Can a hub agent in a star network serve as an effective information sink to improve learning rates for the entire network?
• RQ4 What types of policies lead to high regret in networked collaborative settings despite individual consistency?
• RQ5 To what extent can network topology be leveraged to reduce collective regret in decentralized learning?

Key findings

• Naive extensions of single-agent bandit policies to networked settings can lead to high regret due to non-altruistic behavior.
• A class of non-altruistic and individually consistent policies suffers from poor collective performance in collaborative bandit learning.
• The hub agent in a star network can act as an information sink, significantly improving learning efficiency across the network.
• Exploiting network structure—particularly in star topologies—leads to substantial reductions in collective regret compared to unstructured or naive policy extensions.
• Numerical experiments confirm that structured information sharing via the hub agent results in faster convergence and lower regret.