[Paper Review] Multi-Agent Deep Reinforcement Learning based Spectrum Allocation for D2D Underlay Communications

Zheng Li, Caili Guo|arXiv (Cornell University)|Dec 18, 2019

Advanced MIMO Systems Optimization40 references4 citations

TL;DR

This paper proposes a multi-agent deep reinforcement learning framework, MAAC, for distributed spectrum allocation in D2D underlay communications, enabling cooperation among D2D users without real-time signaling. It achieves low outage probability for cellular links and high sum rate for D2D links, with a scalable variant, NAAC, offering reduced training complexity and strong generalization.

ABSTRACT

Device-to-device (D2D) communication underlay cellular networks is a promising technique to improve spectrum efficiency. In this situation, D2D transmission may cause severe interference to both the cellular and other D2D links, which imposes a great technical challenge to spectrum allocation. Existing centralized schemes require global information, which causes a large signaling overhead. While existing distributed schemes requires frequent information exchange among D2D users and cannot achieve global optimization. In this paper, a distributed spectrum allocation framework based on multi-agent deep reinforcement learning is proposed, named multi-agent actor critic (MAAC). MAAC shares global historical states, actions and policies during centralized training, requires no signal interaction during execution and utilizes cooperation among users to further optimize system performance. Moreover, in order to decrease the computing complexity of the training, we further propose the neighbor-agent actor critic (NAAC) based on the neighbor users' historical information for centralized training. The simulation results show that the proposed MAAC and NAAC can effectively reduce the outage probability of cellular links, greatly improve the sum rate of D2D links and converge quickly.

Motivation & Objective

• Address the challenge of interference in D2D underlay communications where D2D links reuse cellular spectrum.
• Overcome the limitations of centralized schemes (high signaling overhead) and distributed schemes (frequent information exchange) in dense, dynamic networks.
• Design a distributed spectrum allocation framework that enables cooperation among D2D users without real-time signaling.
• Reduce training complexity while maintaining high system performance through centralized training with distributed execution.
• Ensure reliable cellular link performance while maximizing D2D sum rate in dynamic, user-intensive environments.

Proposed method

• Proposes MAAC (Multi-Agent Actor-Critic), a centralized training with distributed execution framework using global historical states, actions, and policies to enable cooperation among D2D agents.
• Introduces NAAC (Neighbor-Agent Actor-Critic), a scalable variant that uses only the historical data of a fixed number of neighboring D2D users during training to reduce computational load.
• Employs deep Q-networks and policy gradient methods within a multi-agent RL framework to learn optimal spectrum allocation policies.
• Designs a reward function that balances cellular link outage probability and D2D sum rate to align learning with system objectives.
• Trains policies in a centralized manner using historical data from all or neighboring agents, then deploys them in a distributed execution mode.
• Uses a parameter sharing mechanism across agents to improve sample efficiency and generalization in multi-agent settings.

Experimental results

Research questions

• RQ1 Can a multi-agent deep reinforcement learning framework effectively reduce interference and improve spectral efficiency in D2D underlay communications without global real-time signaling?
• RQ2 How does using global historical data during centralized training affect the performance and convergence of distributed spectrum allocation?
• RQ3 To what extent can a neighbor-based information aggregation strategy (NAAC) reduce training complexity while maintaining system performance?
• RQ4 How does the number of neighboring users considered during training (λ) impact the outage probability and sum rate of cellular and D2D links?
• RQ5 Can the proposed framework achieve better trade-offs between cellular reliability and D2D throughput compared to existing centralized and distributed schemes?

Key findings

• MAAC reduces the outage probability of cellular links and significantly improves the sum rate of D2D links compared to existing methods.
• NAAC achieves performance comparable to MAAC when λ (number of neighbors considered) is sufficiently large, with a 2 bit/s/Hz lower cellular sum rate at λ=2 but converging to MAAC’s performance as λ increases.
• The D2D sum rate under NAAC is initially higher than MAAC for small λ, but decreases as λ increases due to stricter enforcement of cellular link constraints.
• As λ increases, NAAC’s performance converges to MAAC’s, indicating that incorporating more neighbor history improves policy robustness and reliability.
• Both MAAC and NAAC converge quickly and outperform traditional centralized and distributed schemes in terms of spectral efficiency and reliability.
• The proposed frameworks are well-suited for dense, dynamic networks such as vehicular networks, with low signaling overhead and high scalability.