[Paper Review] Path Design and Resource Management for NOMA enhanced Indoor Intelligent Robots

Ruikang Zhong, Xiao Liu+3 more|arXiv (Cornell University)|Nov 23, 2020

Indoor and Outdoor Localization Technologies4 citations

TL;DR

This paper proposes a NOMA-enhanced indoor intelligent robot (IR) service framework that jointly optimizes path planning and downlink power allocation using a deep transfer deterministic policy gradient (DT-DPG) algorithm trained on a radio map. The radio map, derived from an ITU-compliant indoor channel model, enables efficient, hardware-free training, reducing training time by ~30% and improving reliability and mission efficiency over OMA and standard DDPG approaches.

ABSTRACT

A communication enabled indoor intelligent robots (IRs) service framework is proposed, where non-orthogonal multiple access (NOMA) technique is adopted to enable highly reliable communications. In cooperation with the ultramodern indoor channel model recently proposed by the International Telecommunication Union (ITU), the Lego modeling method is proposed, which can deterministically describe the indoor layout and channel state in order to construct the radio map. The investigated radio map is invoked as a virtual environment to train the reinforcement learning agent, which can save training time and hardware costs. Build on the proposed communication model, motions of IRs who need to reach designated mission destinations and their corresponding down-link power allocation policy are jointly optimized to maximize the mission efficiency and communication reliability of IRs. In an effort to solve this optimization problem, a novel reinforcement learning approach named deep transfer deterministic policy gradient (DT-DPG) algorithm is proposed. Our simulation results demonstrate that 1) With the aid of NOMA techniques, the communication reliability of IRs is effectively improved; 2) The radio map is qualified to be a virtual training environment, and its statistical channel state information improves training efficiency by about 30%; 3) The proposed DT-DPG algorithm is superior to the conventional deep deterministic policy gradient (DDPG) algorithm in terms of optimization performance, training time, and anti-local optimum ability.

Motivation & Objective

• To address the challenge of maintaining reliable, low-latency communication for indoor robots during dynamic path planning.
• To jointly optimize robot trajectory and downlink power allocation under non-orthogonal multiple access (NOMA) to enhance mission success and communication quality.
• To reduce training cost and time for reinforcement learning agents in robotic navigation by using a statistically derived radio map as a virtual environment.
• To improve upon conventional DDPG by introducing transfer learning to avoid local optima and accelerate convergence.

Proposed method

• A radio map is constructed using a Lego modeling method to deterministically represent indoor layout and statistical channel state information based on the ITU’s recent indoor channel model.
• The radio map serves as a virtual training environment for reinforcement learning, replacing real-world random fading with statistical channel data to stabilize training.
• A novel deep transfer deterministic policy gradient (DT-DPG) algorithm is proposed to jointly optimize path planning and NOMA power allocation, with a reward function incorporating mission time, QoS, and outage avoidance.
• Transfer learning is applied by pre-training on a simpler task (e.g., destination reaching) before fine-tuning on the full mission, improving convergence and robustness.
• The reward function includes a base reward for reaching the destination, a penalty for QoS violations, and a term for minimizing total mission time.
• The DT-DPG agent is trained using a replay buffer and target networks, with exploration noise to balance exploitation and exploration.

Experimental results

Research questions

• RQ1 Can a radio map derived from statistical channel state information serve as an effective virtual environment for training RL agents in indoor robotic navigation?
• RQ2 How does the proposed DT-DPG algorithm compare to conventional DDPG in terms of convergence speed, optimization performance, and avoidance of local optima?
• RQ3 To what extent does NOMA improve communication reliability and mission efficiency compared to OMA in indoor robotic networks?
• RQ4 What is the impact of using a radio map on training efficiency and hardware cost reduction in RL-based robot control?

Key findings

• The proposed radio map reduces training time by approximately 30% compared to real-world training by eliminating random fading interference in the neural network convergence process.
• The DT-DPG algorithm achieves a mean mission quality index (MQI) of 929.0 in the NOMA scheme when trained on the radio map, significantly outperforming DDPG, which achieves less than 20% success rate in reaching the destination under the same constraints.
• The DT-DPG algorithm avoids local optima more effectively than standard DDPG due to transfer learning, achieving over 95% success rate in reaching the destination, compared to less than 20% for DDPG.
• NOMA improves MQI by 49.5 points on average compared to OMA (929.0 vs. 879.5) when trained on the radio map, demonstrating superior user fairness and reliability.
• The radio map-trained agent achieves similar performance in real-world testing (MQI = 919.0) to a real-world trained agent (MQI = 929.0), confirming the validity of the virtual environment.
• The multi-stage convergence of DT-DPG shows a performance dip after transfer due to increased exploration, followed by a significant performance gain, indicating effective knowledge transfer and improved policy learning.