[Paper Review] Multi-Tenant Cross-Slice Resource Orchestration: A Deep Reinforcement Learning Approach

Xianfu Chen, Zhifeng Zhao+5 more|arXiv (Cornell University)|Jul 17, 2018

Software-Defined Networks and 5G53 references3 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based approach for multi-tenant cross-slice resource orchestration in RAN-only slicing, where service providers (SPs) compete for channel access to serve their mobile users (MUs). By modeling the problem as a stochastic game and using linear decomposition of per-SP Markov decision processes, the method enables independent, online learning of optimal bidding and offloading policies, achieving significant performance gains over baselines in utility, queue length, and energy efficiency.

ABSTRACT

With the cellular networks becoming increasingly agile, a major challenge lies in how to support diverse services for mobile users (MUs) over a common physical network infrastructure. Network slicing is a promising solution to tailor the network to match such service requests. This paper considers a system with radio access network (RAN)-only slicing, where the physical infrastructure is split into slices providing computation and communication functionalities. A limited number of channels are auctioned across scheduling slots to MUs of multiple service providers (SPs) (i.e., the tenants). Each SP behaves selfishly to maximize the expected long-term payoff from the competition with other SPs for the orchestration of channels, which provides its MUs with the opportunities to access the computation and communication slices. This problem is modelled as a stochastic game, in which the decision makings of a SP depend on the global network dynamics as well as the joint control policy of all SPs. To approximate the Nash equilibrium solutions, we first construct an abstract stochastic game with the local conjectures of channel auction among the SPs. We then linearly decompose the per-SP Markov decision process to simplify the decision makings at a SP and derive an online scheme based on deep reinforcement learning to approach the optimal abstract control policies. Numerical experiments show significant performance gains from our scheme.

Motivation & Objective

• To address the challenge of orchestrating limited radio resources among multiple selfish service providers (SPs) in a shared RAN-only slicing environment.
• To enable efficient cross-slice coordination between computation and communication resources for mobile users (MUs) under dynamic and competitive conditions.
• To design a scalable, decentralized decision-making framework that allows SPs to act independently while approximating Nash equilibrium behavior.
• To reduce dependency on global network state and full knowledge of other SPs’ strategies through conjecture-based abstraction and online DRL.
• To improve system-wide performance in terms of utility, queue length, packet drops, and energy consumption under varying traffic loads and channel availability.

Proposed method

• Formulates the multi-tenant resource orchestration problem as a non-cooperative stochastic game, where each SP aims to maximize its long-term payoff.
• Introduces an abstract stochastic game model that replaces full network state dependence with local conjectures about other SPs’ behaviors.
• Applies linear decomposition to simplify each SP’s Markov decision process (MDP), enabling independent decision-making per SP.
• Develops an online deep reinforcement learning (DRL) scheme to iteratively learn optimal control policies in real time without requiring full system state knowledge.
• Employs a VCG pricing mechanism at the SDN-orchestrator level to regulate channel auctions at the start of each scheduling slot.
• Uses a double-dueling DQN architecture to stabilize training and improve sample efficiency in the DRL component.

Experimental results

Research questions

• RQ1 How can a decentralized, scalable resource orchestration scheme be designed for multi-tenant RAN-only slicing with competing SPs?
• RQ2 What is the impact of incomplete information and strategic behavior among SPs on system performance in a shared RAN environment?
• RQ3 Can a DRL-based approach effectively learn optimal bidding, offloading, and scheduling policies under dynamic traffic and limited channel access?
• RQ4 How does the proposed method compare to baseline strategies in terms of utility, queue stability, and energy efficiency?
• RQ5 To what extent does the linear decomposition of the per-SP MDP improve computational tractability and learning convergence?

Key findings

• The proposed DRL-based scheme achieves significantly higher average utility per MU compared to all three baselines across varying traffic loads and channel counts.
• At a packet arrival rate of λ=8, the proposed scheme reduces average queue length by up to 40% and packet drops by up to 50% compared to Baseline 1.
• With increasing numbers of channels, the proposed scheme reduces average queue length and packet drops more effectively than baselines, showing improved scalability.
• While Baseline 2 reduces CPU energy consumption due to higher offloading rates, the proposed scheme maintains better QoS by minimizing packet drops, even at the cost of slightly higher CPU energy.
• The scheme demonstrates stable convergence over the learning procedure, with consistent performance gains in utility, queue management, and energy efficiency.
• The linear decomposition of the per-SP MDP enables effective independent learning and reduces computational complexity without sacrificing performance.