[Paper Review] Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks

Shangxing Wang, Hanpeng Liu+2 more|arXiv (Cornell University)|Feb 20, 2018

Advanced Wireless Network OptimizationEngineering26 references19 citations

TL;DR

This paper proposes a Deep Q-Network (DQN) framework for dynamic multichannel access in wireless networks with correlated channels and unknown system dynamics. By leveraging deep reinforcement learning, the DQN learns optimal channel selection policies through online interaction without prior knowledge of channel statistics, achieving near-optimal performance in both simulated and real-world trace-based scenarios, including time-varying environments via an adaptive DQN extension.

ABSTRACT

We consider a dynamic multichannel access problem, where multiple correlated channels follow an unknown joint Markov model. A user at each time slot selects a channel to transmit data and receives a reward based on the success or failure of the transmission. The objective is to find a policy that maximizes the expected long-term reward. The problem is formulated as a partially observable Markov decision process (POMDP) with unknown system dynamics. To overcome the challenges of unknown system dynamics as well as prohibitive computation, we apply the concept of reinforcement learning and implement a Deep Q-Network (DQN) that can deal with large state space without any prior knowledge of the system dynamics. We provide an analytical study on the optimal policy for fixed-pattern channel switching with known system dynamics and show through simulations that DQN can achieve the same optimal performance without knowing the system statistics. We compare the performance of DQN with a Myopic policy and a Whittle Index-based heuristic through both simulations as well as real-data trace and show that DQN achieves near-optimal performance in more complex situations. Finally, we propose an adaptive DQN approach with the capability to adapt its learning in time-varying, dynamic scenarios.

Motivation & Objective

• To address dynamic multichannel access in wireless networks where channels are correlated and system dynamics are unknown.
• To overcome the prohibitive computational complexity of solving the underlying POMDP formulation with unknown transition dynamics.
• To develop a reinforcement learning-based solution that learns optimal channel access policies directly from observations and rewards.
• To evaluate performance in complex, real-world scenarios using both synthetic simulations and real radio trace data.
• To design an adaptive DQN framework capable of detecting and re-learning in non-stationary, time-varying environments.

Proposed method

• Formulate the multichannel access problem as a partially observable Markov decision process (POMDP) with a joint Markov model over N correlated channels.
• Implement a Deep Q-Network (DQN) that uses raw channel state observations as input and outputs Q-values for channel selection actions.
• Train the DQN through online interaction with the environment, updating Q-values using experience replay and target networks to stabilize learning.
• Design an adaptive DQN variant that detects environmental shifts and fine-tunes its policy using online retraining.
• Compare DQN performance against a Myopic policy and a Whittle Index-based heuristic in both simulation and real-data trace experiments.
• Use real-world channel activity traces from IEEE 802.15.4-based systems to validate performance under practical interference conditions.

Experimental results

Research questions

• RQ1 Can a DQN agent learn an optimal or near-optimal channel access policy in a multichannel system with unknown, correlated channel dynamics?
• RQ2 How does DQN performance compare to traditional heuristics like Myopic and Whittle Index policies in complex, correlated channel environments?
• RQ3 Can the DQN framework generalize to real-world channel traces with non-i.i.d. and correlated activity patterns?
• RQ4 Can an adaptive DQN detect environmental changes and re-learn optimal policies in non-stationary dynamic scenarios?
• RQ5 What is the scalability and robustness of the DQN approach in multi-user settings with potential interference?

Key findings

• The DQN achieves the same optimal performance as a genie-aided policy in fixed-pattern channel switching scenarios, despite having no prior knowledge of system statistics.
• In complex, correlated channel environments, DQN outperforms both the Myopic and Whittle Index-based heuristics in terms of average discounted reward.
• Through real-data trace evaluation, DQN maintains near-optimal performance even under practical, non-stationary interference patterns from Wi-Fi, Bluetooth, and microwave ovens.
• The adaptive DQN framework successfully detects changes in channel dynamics and re-learns optimal policies in time-varying environments, demonstrating robustness to non-stationarity.
• In multi-user scenarios with up to four users and eight channels, DQN maintains strong performance, indicating scalability to small-scale multi-user settings.
• The authors release the channel models and real-world trace data online to support reproducibility and benchmarking in future research.