[Paper Review] Dynamic Coded Caching in Wireless Networks Using Multi-Agent Reinforcement Learning

Jesper Pedersen, Alexandre Graell i Amat+2 more|arXiv (Cornell University)|Apr 14, 2021

Caching and Content Delivery23 references4 citations

TL;DR

This paper proposes a multi-agent reinforcement learning (MARL) framework for dynamic coded caching in wireless networks, using soft time-to-live (STTL) policies to minimize backhaul load. It demonstrates that MARL outperforms optimized, knowledge-dependent policies—especially under non-uniform spatial request distributions—achieving up to 20% lower network load for C=4 and ζ=0.9.

ABSTRACT

We consider distributed caching of content across several small base stations (SBSs) in a wireless network, where the content is encoded using a maximum distance separable code. Specifically, we apply soft time-to-live (STTL) cache management policies, where coded packets may be evicted from the caches at periodic times. We propose a reinforcement learning (RL) approach to find coded STTL policies minimizing the overall network load. We demonstrate that such caching policies achieve almost the same network load as policies obtained through optimization, where the latter assumes perfect knowledge of the distribution of times between file requests as well the distribution of the number of SBSs within communication range of a user placing a request. We also suggest a multi-agent RL (MARL) framework for the scenario of non-uniformly distributed requests in space. For such a scenario, we show that MARL caching policies achieve lower network load as compared to optimized caching policies assuming a uniform request placement. We also provide convincing evidence that synchronous updates offer a lower network load than asynchronous updates for spatially homogeneous renewal request processes due to the memory of the renewal processes.

Motivation & Objective

• Address the challenge of minimizing backhaul load in distributed coded caching under uncertain or unknown request statistics and SBS distribution.
• Develop a reinforcement learning (RL) approach that eliminates the need for prior knowledge of request process distributions and SBS availability.
• Investigate the impact of synchronous vs. asynchronous cache updates on network load in spatially homogeneous and heterogeneous request processes.
• Design a multi-agent RL (MARL) framework where SBSs autonomously learn caching policies, enabling decentralized and adaptive operation.
• Evaluate the performance of MARL policies against optimized policies that assume perfect knowledge and synchronous updates, particularly in non-uniform request scenarios.

Proposed method

• Formulate the dynamic coded STTL caching problem as a deep deterministic policy gradient (DDPG) reinforcement learning problem with a single agent for synchronous updates.
• Use neural networks to approximate the actor and critic functions, with one-hot encoding of requested files and sigmoid outputs for actions (coded packet eviction probabilities).
• Implement exploration via zero-mean Gaussian noise with variance σ² = 0.01, and use the Adam optimizer with a memory buffer of size 10⁶ and batch size 64.
• For the MARL setup, model each SBS as an independent agent, allowing asynchronous learning and decentralized policy updates without global synchronization.
• Train the DDPG agent with a discount factor γ = 0.99 and target network update rate ρ = 0.999 to stabilize learning.
• Evaluate performance under varying user request spatial distributions, parameterized by ζ, the probability that a user requesting a file is within range of its associated SBS.

Experimental results

Research questions

• RQ1 Can a single-agent DDPG-based RL approach achieve backhaul load reductions comparable to optimized policies that assume perfect knowledge of request statistics and SBS distribution?
• RQ2 Does synchronous cache updating lead to lower network load than asynchronous updating in spatially homogeneous renewal request processes?
• RQ3 How does the performance of MARL-based caching policies compare to optimized policies that assume uniform request distribution and synchronous updates in non-uniform spatial request scenarios?
• RQ4 To what extent does the memory inherent in renewal processes affect the effectiveness of synchronous vs. asynchronous cache updates?
• RQ5 Can MARL enable SBSs to autonomously learn effective caching policies without centralized coordination or prior statistical knowledge?

Key findings

• The single-agent DDPG-based RL approach achieves backhaul load reductions nearly identical to those of optimized policies that require perfect knowledge of request statistics and SBS distribution.
• Synchronous cache updates yield lower network load than asynchronous updates under spatially homogeneous renewal processes, indicating that the memory of the request process enhances performance more than the cost of synchronization.
• For spatially non-uniform request processes, MARL policies reduce network load by up to 20% compared to optimized policies that assume uniformity and synchronous updates, particularly when ζ = 0.9 and cache size C = 4.
• The MARL framework outperforms optimized policies in heterogeneous scenarios, demonstrating that decentralized, autonomous learning adapts better to real-world spatial non-uniformity.
• The performance gain from MARL is most pronounced when the probability of a user being within range of an SBS (ζ) deviates from the uniform case (ζ = π/8 ≈ 0.39), especially for ζ > π/8.
• The DDPG algorithm with exploration noise and batch normalization achieves stable training and effective policy learning without requiring prior statistical assumptions.