[Paper Review] UAV-Aided Cellular Communications with Deep Reinforcement Learning Against Jamming

Xiaozhen Lu, Liang Xiao+2 more|arXiv (Cornell University)|May 17, 2018

UAV Applications and OptimizationEngineering15 references17 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based UAV relay scheme that optimizes relay power in cellular networks under smart jamming without requiring knowledge of the jamming or network model. By leveraging deep Q-networks (DQN), experience replay, and transfer learning, the UAV accelerates learning, reduces bit error rate (BER) by up to 99.7% and saves 33.6% energy compared to prior methods, achieving robust anti-jamming performance in dynamic environments.

ABSTRACT

Cellular systems are vulnerable to jamming attacks, especially smart jammers that choose their jamming policies such as the jamming channel frequencies and power based on the ongoing communication policies and network states. In this article, we present an unmanned aerial vehicle (UAV) aided cellular communication framework against jamming. In this scheme, UAVs use reinforcement learning methods to choose the relay policy for mobile users in cellular systems, if the serving base station is heavily jammed. More specifically, we propose a deep reinforcement learning based UAV relay scheme to help cellular systems resist smart jamming without being aware of the jamming model and the network model in the dynamic game based on the previous anti-jamming relay experiences and the observed current network status. This scheme can achieve the optimal performance after enough interactions with the jammer. Simulation results show that this scheme can reduce the bit error rate of the messages and save energy for the cellular system compared with the existing scheme.

Motivation & Objective

• To address the vulnerability of cellular systems to smart jamming attacks that exploit dynamic network states and adaptive jamming strategies.
• To design a UAV-aided relay system that operates effectively under unknown jamming and network models in real-time dynamic environments.
• To reduce learning time and energy consumption in anti-jamming relay selection by replacing traditional Q-learning with deep reinforcement learning and transfer learning.
• To improve system robustness against state estimation errors and delays common in practical UAV-aided cellular networks.

Proposed method

• The UAV uses a deep Q-network (DQN) with a convolutional neural network (CNN) to compress high-dimensional state space, including BER, channel gains, and jamming power.
• Experience replay is employed to stabilize training and improve sample efficiency by storing and reusing past state-action-reward transitions.
• Transfer learning initializes the CNN weights using prior anti-jamming relay experiences from similar scenarios, accelerating convergence and reducing risky exploration.
• The relay power is selected based on Q-values output by the DQN, which maximizes UAV utility while minimizing BER and energy cost.
• The system formulates the relay decision as a Markov decision process (MDP), modeling interactions between UAV and jammer as a dynamic game.
• The scheme is evaluated under static, reactive, and smart jammers that use RL to minimize UAV utility with low jamming cost.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning-based UAV relay scheme achieve optimal relay power selection without prior knowledge of the jamming or network model?
• RQ2 How does transfer learning improve learning speed and reduce energy consumption in dynamic anti-jamming relay games?
• RQ3 What is the performance gain in BER and energy efficiency compared to existing schemes like HPUR and Q-learning?
• RQ4 How robust is the proposed scheme to state estimation delay and error in practical UAV-aided cellular systems?

Key findings

• The proposed DRLUR scheme reduces the bit error rate (BER) of user messages by 99.7% at the 1000th time slot compared to a Q-learning-based relay algorithm.
• The scheme saves 33.6% in cellular system energy consumption compared to the HPUR benchmark, with a 24.6% reduction in energy use at time slot 1500.
• DRLUR converges to the Nash equilibrium of the theoretical game model in long-duration dynamic relay games, indicating stable optimal performance.
• The learning time is reduced by 84.6%—DRLUR optimizes relay power in about 200 time slots (0.22s), compared to significantly longer times for HPUR.
• The scheme is less sensitive to state estimation delay and error, maintaining low BER even with Gaussian-distributed estimation errors (σ = 1.5).
• The BER performance bound is analytically derived and validated through simulations, confirming the scheme’s reliability under varying jamming conditions.