[Paper Review] Decentralized Learning for Channel Allocation in IoT Networks over Unlicensed Bandwidth as a Contextual Multi-player Multi-armed Bandit Game

Wenbo Wang, Amir Leshem+2 more|arXiv (Cornell University)|Mar 30, 2020

Advanced Bandit Algorithms Research42 references18 citations

TL;DR

This paper proposes a decentralized, three-stage trial-and-error learning algorithm for channel allocation in unlicensed IoT networks, modeling the problem as a contextual multi-player multi-armed bandit game. It achieves sub-linear (polylogarithmic) regret and converges to social-optimal channel allocation without explicit coordination, leveraging only local collision feedback and context-aware learning to balance efficiency and scalability in dynamic, interference-prone environments.

ABSTRACT

We study a decentralized channel allocation problem in an ad-hoc Internet of Things network underlaying on the spectrum licensed to a primary cellular network. In the considered network, the impoverished channel sensing/probing capability and computational resource on the IoT devices make them difficult to acquire the detailed Channel State Information (CSI) for the shared multiple channels. In practice, the unknown patterns of the primary users' transmission activities and the time-varying CSI (e.g., due to small-scale fading or device mobility) also cause stochastic changes in the channel quality. Decentralized IoT links are thus expected to learn channel conditions online based on partial observations, while acquiring no information about the channels that they are not operating on. They also have to reach an efficient, collision-free solution of channel allocation with limited coordination. Our study maps this problem into a contextual multi-player, multi-armed bandit game, and proposes a purely decentralized, three-stage policy learning algorithm through trial-and-error. Theoretical analyses shows that the proposed scheme guarantees the IoT links to jointly converge to the social optimal channel allocation with a sub-linear (i.e., polylogarithmic) regret with respect to the operational time. Simulations demonstrate that it strikes a good balance between efficiency and network scalability when compared with the other state-of-the-art decentralized bandit algorithms.

Motivation & Objective

• To address the challenge of decentralized, efficient channel allocation in ad-hoc IoT networks underlaying licensed cellular bands with limited sensing and computational capabilities.
• To enable IoT devices to learn optimal channel associations in real time using only partial observations and local feedback, without knowledge of unused channels or global coordination.
• To achieve social-optimal performance—maximizing total network throughput—while minimizing collisions and channel switching in time-varying, interference-prone environments.
• To design a scalable, lightweight solution suitable for infrastructure-less, power-constrained IoT deployments.

Proposed method

• Models the channel allocation problem as a contextual multi-player multi-armed bandit (MP-MAB) game, where IoT devices are players, channels are arms, and rewards depend on both channel state and primary user activity (context).
• Introduces a three-stage, epoch-based policy learning framework: (1) channel quality exploration via trial-and-error, (2) intermediate non-cooperative game formation for decentralized policy learning, and (3) policy exploitation with minimal coordination.
• Uses local feedback from receiving devices to detect inter-link collisions, enabling devices to infer channel quality and adjust strategies without explicit signaling.
• Employs a context-aware reward model where arm values (channel quality) are dynamically influenced by the presence and activity of primary users.
• Applies a perturbation-based learning phase to explore policies across different contexts, improving adaptability to time-varying interference and fading.
• Theoretical analysis proves convergence to social optimality with regret bounded by O(M log^{1+δ}_2 T) for M players over time horizon T (δ > 0).

Experimental results

Research questions

• RQ1 Can a purely decentralized, lightweight algorithm achieve social-optimal channel allocation in unlicensed IoT networks with no coordination or global channel state information?
• RQ2 How can IoT devices learn optimal channel associations in real time when they cannot sense or estimate all channels simultaneously due to resource constraints?
• RQ3 What is the impact of primary user activity and time-varying channel conditions on the performance of decentralized learning algorithms in multi-channel IoT networks?
• RQ4 How does context-aware learning improve performance compared to non-contextual MP-MAB algorithms in terms of regret, collisions, and scalability?
• RQ5 Can the proposed algorithm maintain high efficiency and scalability as the number of IoT devices increases, especially in the absence of infrastructure?

Key findings

• The proposed algorithm achieves the best average sum of normalized rates among all compared algorithms (MC, SOC, GoT) in simulations, demonstrating superior network efficiency over time.
• Despite slightly higher collision counts due to frequent policy exploration, the algorithm converges faster than GoT, which suffers from excessive collisions due to longer policy-learning phases.
• The algorithm exhibits lower channel switching frequency than non-contextual algorithms (MC, SOC), indicating higher policy consistency across different contexts, which enhances stability and adaptability.
• In large-scale networks (up to 30 nodes), the proposed algorithm maintains better performance than MC and SOC, with a manageable increase in collisions due to extended learning phases.
• Theoretical analysis confirms that the regret grows sub-linearly as O(M log^{1+δ}_2 T), proving the algorithm’s long-term efficiency and convergence to social optimality.
• The algorithm scales well with network size, outperforming GoT significantly in larger networks, where GoT’s performance degrades due to longer learning phases and larger auxiliary state spaces.