[Paper Review] Band-limited Soft Actor Critic Model

Miguel Vázquez-Martin del Campo, Zhengxing Chen+3 more|arXiv (Cornell University)|Jun 19, 2020

Neural Networks and Reservoir Computing19 references4 citations

TL;DR

This paper proposes a band-limited Soft Actor-Critic (SAC) model that improves sample efficiency and training stability by applying a separable convolutional filter to constrain the critic's spatial resolution, reducing interference from high-frequency components in the value function. The method enhances performance in continuous control environments, particularly under noisy real-world conditions, by decoupling low- and high-frequency components of the state-action value approximation.

ABSTRACT

Soft Actor Critic (SAC) algorithms show remarkable performance in complex simulated environments. A key element of SAC networks is entropy regularization, which prevents the SAC actor from optimizing against fine grained features, oftentimes transient, of the state-action value function. This results in better sample efficiency during early training. We take this idea one step further by artificially bandlimiting the target critic spatial resolution through the addition of a convolutional filter. We derive the closed form solution in the linear case and show that bandlimiting reduces the interdependency between the low and high frequency components of the state-action value approximation, allowing the critic to learn faster. In experiments, the bandlimited SAC outperformed the classic twin-critic SAC in a number of Gym environments, and displayed more stability in returns. We derive novel insights about SAC by adding a stochastic noise disturbance, a technique that is increasingly being used to learn robust policies that transfer well to the real world counterparts.

Motivation & Objective

• To investigate how spectral bias and frequency components in the state-action value function affect SAC's training dynamics and performance.
• To address the instability and variance introduced by high-frequency components in the critic's value function approximation, which can propagate to low-frequency components and hinder policy convergence.
• To explore whether explicitly bandlimiting the critic's output improves learning efficiency and robustness, especially in noisy or imperfect real-world environments.
• To understand the role of entropy regularization and temperature hyperparameters in coupling low- and high-frequency components of the value function.

Proposed method

• Introduces a separable convolutional filter to bandlimit the target critic's output, restricting its ability to learn high-frequency components of the state-action value function.
• Matches the filter bandwidth to the policy's effective resolution, dynamically adjusting to the policy's entropy and exploration behavior.
• Derives a closed-form solution for the bandlimited critic in the linear case, showing complete decoupling of low- and high-frequency components under uniform policies.
• Employs an Ornstein-Uhlenbeck process to simulate self-correlated temporal and spatial noise in the reward function, emulating real-world disturbances.
• Compares bandlimited SAC against standard SAC in Gym environments under both noiseless and noisy reward conditions, with fixed temperature to isolate the effect of bandlimiting.
• Uses t-statistics to evaluate the significance of performance differences in mean returns between the two models.

Experimental results

Research questions

• RQ1 Does bandlimiting the critic's spatial resolution improve sample efficiency and asymptotic performance in continuous control environments?
• RQ2 How does bandlimiting affect the interdependency between low- and high-frequency components of the state-action value function?
• RQ3 Can bandlimiting enhance robustness to self-correlated noise in the reward function, simulating real-world environmental disturbances?
• RQ4 What is the relationship between the temperature hyperparameter and the degree of coupling between low- and high-frequency components in the value function?
• RQ5 Does bandlimiting lead to more stable learning trajectories and reduced variance in return during late training stages?

Key findings

• The bandlimited SAC outperformed standard SAC in three out of five Gym environments in terms of mean return, with statistically significant improvements (t-statistic > 1) in Ant, HalfCheetah, and Hopper.
• Bandlimiting improved training stability, as evidenced by more consistent returns and reduced variance in performance across runs.
• In noisy environments with self-correlated reward disturbances, the bandlimited SAC showed higher returns than standard SAC, even when temperature was fixed, indicating reduced sensitivity to high-frequency noise.
• The closed-form solution revealed that low- and high-frequency components are completely decoupled in the value function approximation under uniform policies, with coupling diminishing as policies become deterministic.
• The temperature hyperparameter in bandlimited SAC plays a dual role: regulating policy entropy and controlling the degree of interdependency between frequency components in the value function.
• The results suggest that high-frequency components of the value function are harder to learn and can introduce instability, and that constraining them improves overall learning efficiency and robustness.