[Paper Review] Deep Reinforcement Learning for Multi-Agent Power Control in Heterogeneous Networks

Lin Zhang, Ying‐Chang Liang|arXiv (Cornell University)|Apr 25, 2020

Full-Duplex Wireless Communications39 references4 citations

TL;DR

This paper proposes a multi-agent deep reinforcement learning (DRL) framework called Multiple-Actor-Shared-Critic (MASC) for power control in heterogeneous networks (HetNets), where each access point (AP) acts as an independent agent with a local deep neural network (DNN). The MASC method enables online, decentralized training using local observations and a shared global critic, achieving higher sum-rate and lower computational latency than conventional WMMSE and fractional programming methods, especially in dynamic, fast-fading environments with limited global CSI availability.

ABSTRACT

We consider a typical heterogeneous network (HetNet), in which multiple access points (APs) are deployed to serve users by reusing the same spectrum band. Since different APs and users may cause severe interference to each other, advanced power control techniques are needed to manage the interference and enhance the sum-rate of the whole network. Conventional power control techniques first collect instantaneous global channel state information (CSI) and then calculate sub-optimal solutions. Nevertheless, it is challenging to collect instantaneous global CSI in the HetNet, in which global CSI typically changes fast. In this paper, we exploit deep reinforcement learning (DRL) to design a multi-agent power control algorithm in the HetNet. To be specific, by treating each AP as an agent with a local deep neural network (DNN), we propose a multiple-actor-shared-critic (MASC) method to train the local DNNs separately in an online trial-and-error manner. With the proposed algorithm, each AP can independently use the local DNN to control the transmit power with only local observations. Simulations results show that the proposed algorithm outperforms the conventional power control algorithms in terms of both the converged average sum-rate and the computational complexity.

Motivation & Objective

• To address the challenge of dynamic, fast-fading wireless channels in HetNets where global instantaneous CSI is hard to collect and conventional power control algorithms become outdated.
• To design a decentralized, real-time power control mechanism that enables each AP to optimize its transmit power using only local information.
• To overcome the computational delay and sub-optimality of traditional algorithms like WMMSE and FP in time-varying environments.
• To develop a scalable, online training framework for multi-agent DRL that maintains global performance while allowing local autonomy.
• To demonstrate that DRL can achieve near-optimal sum-rate performance without requiring global CSI or centralized coordination.

Proposed method

• Each access point (AP) is modeled as an independent agent with a local DNN that maps local channel state information (CSI) to transmit power (action).
• A Multiple-Actor-Shared-Critic (MASC) framework is proposed, consisting of multiple actor DNNs (one per AP) and a single shared critic DNN in the core network.
• The shared critic DNN evaluates the global impact of each AP's action based on historical global state transitions, providing a centralized reward signal for policy learning.
• Each actor DNN is trained online through trial-and-error using the critic's evaluation, enabling decentralized yet globally coordinated policy optimization.
• The method avoids the need for global CSI during inference and reduces computational overhead by replacing iterative optimization with fast DNN inference.
• The framework is trained end-to-end using deep Q-learning or similar DRL algorithms, with experience replay and target networks for stability.

Experimental results

Research questions

• RQ1 Can a decentralized DRL-based power control framework achieve higher sum-rate than conventional centralized algorithms like WMMSE and FP in dynamic HetNets?
• RQ2 To what extent can a multi-agent DRL approach with local observations and a shared critic outperform global-CSI-dependent algorithms in fast-fading environments?
• RQ3 How does the proposed MASC framework reduce computational complexity compared to iterative optimization methods like WMMSE and FP?
• RQ4 Does the MASC method maintain high performance under varying channel correlation and network topology (e.g., two-layer vs. three-layer HetNet)?
• RQ5 Can the DRL-based approach generalize to unseen channel conditions without retraining, given its reliance on local observations only?

Key findings

• The proposed MASC algorithm converges to a sum-rate performance that exceeds both WMMSE and FP algorithms after approximately 2,500 time slots in a three-layer HetNet with i.i.d. fading.
• In the testing phase, the MASC algorithm achieves a higher average sum-rate than WMMSE and FP, particularly in dynamic environments with random channel correlation factors (ρ).
• The average computational latency for power allocation using a local DNN is only 0.34 ms, significantly lower than WMMSE (120 ms) and FP (79 ms).
• Training the critic DNN and actor DNNs takes less than 10 milliseconds on average, indicating feasibility for real-time deployment in practical networks.
• The performance advantage is more pronounced in two-layer HetNets than in three-layer HetNets, due to increased topological complexity and difficulty in policy learning.
• The MASC framework maintains high performance even with random channel correlation (ρ), demonstrating robustness to channel variation and reduced dependency on perfect global CSI.