[Paper Review] Caching Transient Content for IoT Sensing: Multi-Agent Soft Actor-Critic

Xiongwei Wu, Xiuhua Li+4 more|arXiv (Cornell University)|Aug 30, 2020

Age of Information Optimization45 references4 citations

TL;DR

This paper proposes a multi-agent soft actor-critic (MADDPG-SAC) reinforcement learning framework for caching transient IoT sensing data at edge nodes, optimizing for age of information (AoI) while minimizing energy and fronthaul costs. The method achieves superior performance over DQN and traditional actor-critic baselines, especially as network scale increases, with a decentralized variant maintaining near-centralized performance.

ABSTRACT

Edge nodes (ENs) in Internet of Things commonly serve as gateways to cache sensing data while providing accessing services for data consumers. This paper considers multiple ENs that cache sensing data under the coordination of the cloud. Particularly, each EN can fetch content generated by sensors within its coverage, which can be uploaded to the cloud via fronthaul and then be delivered to other ENs beyond the communication range. However, sensing data are usually transient with time whereas frequent cache updates could lead to considerable energy consumption at sensors and fronthaul traffic loads. Therefore, we adopt age of information to evaluate data freshness and investigate intelligent caching policies to preserve data freshness while reducing cache update costs. Specifically, we model the cache update problem as a cooperative multi-agent Markov decision process with the goal of minimizing the long-term average weighted cost. To efficiently handle the exponentially large number of actions, we devise a novel reinforcement learning approach, which is a discrete multi-agent variant of soft actor-critic (SAC). Furthermore, we generalize the proposed approach into a decentralized control, where each EN can make decisions based on local observations only. Simulation results demonstrate the superior performance of the proposed SAC-based caching schemes.

Motivation & Objective

• Address the challenge of caching transient sensing data in IoT networks where data freshness degrades over time.
• Reduce energy consumption and fronthaul traffic from frequent cache updates while maintaining low age of information (AoI).
• Develop a cooperative multi-agent Markov decision process (MMDP) framework to jointly optimize AoI, energy, and fronthaul costs.
• Design a low-complexity, discrete multi-agent reinforcement learning algorithm suitable for large-scale edge caching.
• Generalize the solution to a decentralized control architecture where each edge node acts based on local observations only.

Proposed method

• Model the cache update problem as a cooperative multi-agent Markov decision process (MMDP) with a weighted cost objective combining AoI, energy, and fronthaul load.
• Propose a discrete variant of soft actor-critic (SAC) tailored for multi-agent, discrete action spaces to reduce space complexity.
• Introduce a stochastic action selection mechanism using Gumbel-Softmax reparameterization to enable differentiable exploration.
• Derive a closed-form expression for expected throughput based on Rayleigh fading and SNR thresholds to model fronthaul reliability.
• Generalize the centralized policy into a decentralized control scheme where each edge node uses only local observations to make decisions.
• Use a weighted cost function with tunable hyperparameters ω₁ and ω₂ to balance AoI and update costs in practice.

Experimental results

Research questions

• RQ1 How can we jointly optimize data freshness (AoI), energy consumption, and fronthaul traffic in IoT edge caching?
• RQ2 What reinforcement learning method can efficiently handle the exponential action space in discrete multi-agent edge caching?
• RQ3 Can a decentralized multi-agent SAC approach achieve performance close to centralized learning in large-scale IoT networks?
• RQ4 How do the trade-offs between AoI, energy, and fronthaul load vary with different cost function weights (ω₁, ω₂)?
• RQ5 What is the performance gain of the proposed MADDPG-SAC over DQN and traditional actor-critic baselines in dynamic IoT caching scenarios?

Key findings

• The proposed MADDPG-SAC algorithm significantly outperforms DQN and traditional actor-critic methods in reducing long-term average weighted cost as the number of edge nodes or sensors increases.
• The decentralized variant of the proposed scheme achieves performance within 5% of the centralized version, demonstrating strong scalability and practicality.
• Simulation results show that tuning ω₁ and ω₂ allows effective trade-off between AoI and update costs, with diminishing returns when weights exceed 10.
• The method achieves lower average AoI and reduced energy and fronthaul costs compared to age-optimal and random caching schemes.
• The analytical derivation of expected throughput under Rayleigh fading supports accurate modeling of fronthaul reliability in the cost function.
• The Gumbel-Softmax reparameterization enables effective exploration in discrete action spaces, improving policy convergence and stability.