[Paper Review] ANS: Adaptive Network Scaling for Deep Rectifier Reinforcement Learning Models

Yueh-Hua Wu, Fan-Yun Sun+2 more|arXiv (Cornell University)|Sep 6, 2018

Reinforcement Learning in Robotics22 references3 citations

TL;DR

This paper proposes Adaptive Network Scaling (ANS), a method that dynamically adjusts reward scaling during training to improve performance of deep ReLU-based reinforcement learning agents. ANS enables ReLU networks to automatically discover optimal reward scales, significantly outperforming fixed scaling and alternative activation functions like ELU and sigmoid in actor-critic models.

ABSTRACT

This work provides a thorough study on how reward scaling can affect performance of deep reinforcement learning agents. In particular, we would like to answer the question that how does reward scaling affect non-saturating ReLU networks in RL? This question matters because ReLU is one of the most effective activation functions for deep learning models. We also propose an Adaptive Network Scaling framework to find a suitable scale of the rewards during learning for better performance. We conducted empirical studies to justify the solution.

Motivation & Objective

• To investigate how reward scaling affects deep ReLU-based reinforcement learning models, especially in relation to the dying ReLU problem.
• To determine whether non-saturating activations like ReLU benefit from larger reward scales, contrary to intuition from saturating activations.
• To develop a method that automatically finds optimal reward scaling without retraining, improving sample efficiency and final performance.
• To evaluate the impact of reward scaling on different activation functions (ReLU, ELU, Leaky-ReLU, sigmoid, tanh) in actor-critic frameworks.

Proposed method

• Proposes the Adaptive Network Scaling (ANS) framework, which dynamically adjusts the reward scale during training based on cumulative reward improvement.
• Uses a reward scaling factor that increases multiplicatively when cumulative returns improve and decreases when they stagnate, with configurable parameters $ c_{\text{inc}} $ and $ c_{\text{dec}} $.
• Integrates ANS into actor-critic algorithms (DDPG and A2C) without modifying the network architecture or requiring prior knowledge of optimal scale.
• Employs a tolerance threshold $ T $ to determine when to adjust the scale, based on performance over the last $ T $ episodes or frames.
• Monitors the proportion of dying ReLU units (pseudo-dying ReLU) as a proxy for activation saturation and training stability.
• Uses a decay mechanism with $ \beta = 0.9 $ to smooth the scaling factor update and prevent oscillations.

Experimental results

Research questions

• RQ1 How does reward scaling affect the performance of ReLU-based deep reinforcement learning models, and does it differ from the behavior of saturating activations like sigmoid or tanh?
• RQ2 What is the relationship between reward scaling and the dying ReLU problem in deep RL agents?
• RQ3 How do non-saturating activations like Leaky-ReLU and ELU respond to different reward scales in RL training?
• RQ4 Can an adaptive mechanism be designed to automatically discover the optimal reward scale during training without retraining?

Key findings

• ANS significantly improves performance: DDPG with ANS achieves 9393.24 average return on HalfCheetah-v2, outperforming the original ReLU DDPG (7944.54) and even ELU (7899.17).
• A2C with ANS achieves 3689.64 on HalfCheetah-v2, surpassing all other activation functions and Pop-Art (which yields -455.57).
• ReLU networks with ANS outperform models using ELU, sigmoid, and tanh even without fixed reward scaling, demonstrating superior sample efficiency and final performance.
• The optimal reward scale for ReLU is often larger than expected—ANS typically increases the scale up to 64 before decreasing, indicating that larger scales improve learning.
• The proportion of pseudo-dying ReLU units correlates strongly with both reward scale and final cumulative return, validating it as a useful diagnostic metric.
• Pop-Art, which enforces zero mean and unit variance, performs worse than ANS and even the original ReLU model, suggesting it is suboptimal for actor-critic methods with ReLU.