[Paper Review] Federated Deep Reinforcement Learning for Resource Allocation in O-RAN Slicing

Han Zhang, Hao Zhou+1 more|arXiv (Cornell University)|Aug 2, 2022

Energy Harvesting in Wireless Networks4 citations

TL;DR

This paper proposes a federated deep reinforcement learning (Federated DRL) framework to coordinate two independent xAPPs—power control and slice-based resource allocation—in Open RAN network slicing. By aggregating local Q-tables via a coordination model to form a global Q-table, the method enhances learning efficiency and network performance, achieving 11% higher eMBB throughput and 33% lower URLLC delay compared to independent DRL, with performance close to centralized training.

ABSTRACT

Recently, open radio access network (O-RAN) has become a promising technology to provide an open environment for network vendors and operators. Coordinating the x-applications (xAPPs) is critical to increase flexibility and guarantee high overall network performance in O-RAN. Meanwhile, federated reinforcement learning has been proposed as a promising technique to enhance the collaboration among distributed reinforcement learning agents and improve learning efficiency. In this paper, we propose a federated deep reinforcement learning algorithm to coordinate multiple independent xAPPs in O-RAN for network slicing. We design two xAPPs, namely a power control xAPP and a slice-based resource allocation xAPP, and we use a federated learning model to coordinate two xAPP agents to enhance learning efficiency and improve network performance. Compared with conventional deep reinforcement learning, our proposed algorithm can achieve 11% higher throughput for enhanced mobile broadband (eMBB) slices and 33% lower delay for ultra-reliable low-latency communication (URLLC) slices.

Motivation & Objective

• To address the challenge of coordinating multiple independent xAPP agents with different action and state spaces in O-RAN network slicing.
• To improve learning efficiency and network performance in distributed, privacy-preserving environments where data cannot be centrally shared.
• To enable coordination between heterogeneous xAPPs—power control and resource allocation—without requiring centralized control or global state visibility.
• To achieve performance close to centralized training while maintaining data privacy and reducing training resource consumption.
• To scale the coordination framework to more than two xAPPs in a distributed, federated learning architecture.

Proposed method

• Two distinct xAPPs are designed: a power control xAPP and a hierarchical radio resource allocation xAPP, each operating as an independent reinforcement learning agent.
• Each xAPP trains a local deep Q-network (DQN) model using its own local data and state-action observations.
• Local Q-tables from both xAPPs are submitted to a central coordination model for federated aggregation into a global Q-table.
• The global Q-table is used to select joint actions, which are then decomposed and applied back to the individual xAPPs for execution.
• The framework uses parameter aggregation in a federated manner, preserving data privacy while enabling collaborative learning across heterogeneous agents.
• The coordination model enables information sharing across agents with different state and action spaces, improving overall system performance.

Experimental results

Research questions

• RQ1 Can federated deep reinforcement learning effectively coordinate multiple heterogeneous xAPPs with different state and action spaces in O-RAN network slicing?
• RQ2 How does the performance of federated DRL compare to independent and centralized DRL in terms of throughput and delay for eMBB and URLLC slices?
• RQ3 To what extent does the federated approach improve learning convergence speed and scalability in distributed O-RAN environments?
• RQ4 Can the federated framework maintain data privacy while achieving performance comparable to centralized training?
• RQ5 What is the impact of federated coordination on QoS metrics such as delay and throughput under varying traffic loads?

Key findings

• The proposed federated DRL framework achieves 11% higher throughput for eMBB slices compared to independent DRL under a 10 Mbps eMBB traffic load.
• The framework reduces URLLC slice delay by 33% compared to independent DRL when the eMBB load is 10 Mbps.
• Federated DRL achieves network performance very close to centralized DRL, demonstrating strong coordination capability despite decentralized training.
• The convergence speed of the federated DRL algorithm is faster than centralized DRL, attributed to the reduced state and action space per agent.
• The ECCDF of URLLC delay shows that the federated DRL has a better delay distribution than independent DRL, with lower tail latency.
• The framework is scalable to more than two xAPPs, with computational cost equivalent to training a single DNN with input/output dimensions equal to the sum of all xAPPs' action spaces.