[Paper Review] Deep Reinforcement Learning for IoT Networks: Age of Information and Energy Cost Tradeoff

Xiongwei Wu, Xiuhua Li+3 more|arXiv (Cornell University)|Oct 23, 2020

Age of Information Optimization17 references4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based online cache update framework for IoT networks that optimizes the tradeoff between Age of Information (AoI) and transmission energy cost. By modeling the problem as a Markov decision process and using a deep Q-network (DQN) with a reward function combining AoI and energy cost, the framework learns intelligent caching policies without prior knowledge of content popularity, achieving a 52.7% reduction in energy consumption with only a 2.41-epoch increase in average AoI compared to AoI-optimized settings.

ABSTRACT

In most Internet of Things (IoT) networks, edge nodes are commonly used as to relays to cache sensing data generated by IoT sensors as well as provide communication services for data consumers. However, a critical issue of IoT sensing is that data are usually transient, which necessitates temporal updates of caching content items while frequent cache updates could lead to considerable energy cost and challenge the lifetime of IoT sensors. To address this issue, we adopt the Age of Information (AoI) to quantify data freshness and propose an online cache update scheme to obtain an effective tradeoff between the average AoI and energy cost. Specifically, we first develop a characterization of transmission energy consumption at IoT sensors by incorporating a successful transmission condition. Then, we model cache updating as a Markov decision process to minimize average weighted cost with judicious definitions of state, action, and reward. Since user preference towards content items is usually unknown and often temporally evolving, we therefore develop a deep reinforcement learning (DRL) algorithm to enable intelligent cache updates. Through trial-and-error explorations, an effective caching policy can be learned without requiring exact knowledge of content popularity. Simulation results demonstrate the superiority of the proposed framework.

Motivation & Objective

• To address the challenge of maintaining data freshness in IoT networks while minimizing energy consumption from frequent cache updates.
• To develop an online caching policy that adapts to time-varying user preferences and dynamic wireless channel conditions.
• To model transmission energy consumption realistically, incorporating successful transmission constraints based on SNR thresholds.
• To minimize a weighted sum of average AoI and transmission energy cost through intelligent, adaptive decision-making.
• To enable learning-based cache update without requiring prior knowledge of content popularity dynamics.

Proposed method

• Formulate the cache update problem as a Markov decision process (MDP) with state, action, and reward defined based on AoI, energy cost, and user request statistics.
• Develop a realistic energy consumption model that accounts for successful transmission only when SNR exceeds a threshold, reflecting real wireless channel behavior.
• Design a DQN-based algorithm that learns a caching policy through trial-and-error interactions with the environment, using experience replay and target networks for stability.
• Define the reward function as the negative of the weighted sum of average AoI and transmission energy cost, enabling joint optimization of freshness and energy efficiency.
• Use a replay buffer with a maximum capacity of 5000 experiences and update the target network every 100 training epochs to improve learning convergence.
• Incorporate a hyperparameter η to balance the tradeoff between AoI and energy cost, allowing for adjustable prioritization of either metric.

Experimental results

Research questions

• RQ1 How can a caching policy be designed to balance data freshness (measured by AoI) and energy cost in dynamic IoT networks?
• RQ2 What is the impact of realistic wireless transmission constraints—specifically SNR-based success conditions—on energy consumption modeling in IoT caching?
• RQ3 Can a DRL-based approach learn effective caching policies without prior knowledge of content popularity dynamics?
• RQ4 How does the proposed DRL framework compare to heuristic baselines (e.g., Most Popular Update, Random Update) in terms of energy efficiency and AoI performance?
• RQ5 What is the optimal tradeoff point between AoI and energy cost, and how does it vary with the weighting parameter η?

Key findings

• The proposed DQN algorithm reduces average transmission energy consumption by 52.7% compared to an AoI-optimized baseline (η = 0), with only a 2.41-epoch increase in average AoI.
• The DQN algorithm achieves a 54.9% lower average weighted cost than the Most Popular Update (MPU) baseline when η = 20, demonstrating strong performance gains.
• The learning curve shows that DQN converges to a higher reward level than MPU and RU, achieving 28.8% and 40.4% better performance, respectively, and approaches within 0.84 of the Oracle Update upper bound.
• As η increases, energy cost decreases significantly for DQN and Oracle Update, while AoI increases due to reduced update frequency, confirming the tradeoff behavior.
• The framework effectively adapts to dynamic user preferences and heterogeneous channel conditions, outperforming baselines across all η values, indicating robustness and adaptability.
• The results validate that incorporating physical-layer transmission energy models with SNR constraints leads to more realistic and effective caching policies than simple transmission-count-based models.