[Paper Review] Transfer Learning in Multi-Agent Reinforcement Learning with Double Q-Networks for Distributed Resource Sharing in V2X Communication

Hammad Zafar, Zoran Utkovski+2 more|arXiv (Cornell University)|Jul 13, 2021

Transportation and Mobility Innovations10 references4 citations

TL;DR

This paper proposes a transfer learning-enhanced Double DQN framework for decentralized spectrum sharing in V2X networks, combining Double Q-learning to reduce value overestimation and transfer learning to accelerate training. The method achieves up to 12% higher V2V delivery rates and 6% better V2I sum capacity under limited training time, with median delivery times reduced from 35ms to 18ms.

ABSTRACT

This paper addresses the problem of decentralized spectrum sharing in vehicle-to-everything (V2X) communication networks. The aim is to provide resource-efficient coexistence of vehicle-to-infrastructure(V2I) and vehicle-to-vehicle(V2V) links. A recent work on the topic proposes a multi-agent reinforcement learning (MARL) approach based on deep Q-learning, which leverages a fingerprint-based deep Q-network (DQN) architecture. This work considers an extension of this framework by combining Double Q-learning (via Double DQN) and transfer learning. The motivation behind is that Double Q-learning can alleviate the problem of overestimation of the action values present in conventional Q-learning, while transfer learning can leverage knowledge acquired by an expert model to accelerate learning in the MARL setting. The proposed algorithm is evaluated in a realistic V2X setting, with synthetic data generated based on a geometry-based propagation model that incorporates location-specific geographical descriptors of the simulated environment(outlines of buildings, foliage, and vehicles). The advantages of the proposed approach are demonstrated via numerical simulations.

Motivation & Objective

• Address the challenge of overestimation in Q-values during multi-agent deep reinforcement learning (MARL) for V2X spectrum sharing.
• Reduce training time in MARL-based decentralized resource allocation by leveraging knowledge from a pre-trained expert model.
• Improve system performance in dynamic V2X environments with varying channel conditions and network topologies.
• Evaluate the proposed method in a realistic, geometry-based V2V propagation environment to ensure spatial and temporal consistency.

Proposed method

• Integrate Double DQN (DDQN) to mitigate Q-value overestimation by decoupling action selection from value estimation.
• Apply transfer learning by initializing the learner model’s Q-network with pre-trained weights from an expert model trained via DDQN.
• Use identical state, action, and reward spaces for both expert and learner models to ensure consistent policy transfer.
• Train the expert model for 3,000 episodes and the learner model for only 1,800 episodes to evaluate sample efficiency.
• Employ the GEMV² geometry-based channel model to generate realistic, time-evolving propagation conditions with geographical descriptors.
• Use a fingerprint-based DQN architecture to enable distributed implementation across V2V agents.

Experimental results

Research questions

• RQ1 Can Double DQN reduce Q-value overestimation in MARL-based V2X spectrum sharing, leading to improved policy quality?
• RQ2 Does transfer learning from a pre-trained expert model significantly reduce the required training duration for the learner model in V2X MARL?
• RQ3 How does the performance of the proposed DDQN-TQL method compare to standard DDQN and random baseline under varying payload sizes and channel conditions?
• RQ4 What is the impact of limited training duration on the performance of transfer learning versus training from scratch in V2X MARL?

Key findings

• The DDQN-TQL method reduces median V2V packet delivery time from 35ms (DDQN) to 18ms under a 1,800-episode training limit.
• For a payload size of 4×1060 bytes, DDQN-TQL achieves a 12% improvement in V2V delivery rate and a 6% gain in V2I sum capacity compared to DDQN.
• DDQN-TQL outperforms standard DDQN when training is limited to 1,800 episodes, demonstrating superior sample efficiency.
• For longer training durations (e.g., >3,000 episodes), DDQN without transfer learning outperforms DDQN-TQL, indicating potential overfitting in the transfer learning setup.
• The proposed method shows significant performance gains in high-payload scenarios, where training efficiency and policy quality are most critical.
• The use of the GEMV² model enables realistic, consistent evaluation of MARL policies across time and space, validating the robustness of the proposed framework.