[Paper Review] Distributional Advantage Actor-Critic

Shangda Li, Selina Bing+1 more|arXiv (Cornell University)|Jun 10, 2018

Reinforcement Learning in Robotics4 references7 citations

TL;DR

This paper proposes a Distributional Advantage Actor-Critic (DAAC) approach for first-person navigation in visually complex, partially observable, and procedurally generated environments using raw pixel inputs. By combining distributional RL, advantage function estimation, and hindsight experience replay (HER), the method improves sample efficiency and performance in sparse-reward settings, outperforming standard DQN and DDPG baselines on DeepMind Lab's 'SeekAvoid_Arena' and 'Maze Navigation' tasks.

ABSTRACT

In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a state s and action a, the corresponding value is the expected discounted sum of rewards. The optimal action is then chosen to be the action a with the largest value estimated by value function. However, recent developments have shown both theoretical and experimental evidence of superior performance when value function is replaced with value distribution in context of deep Q learning [1]. In this paper, we develop a new algorithm that combines advantage actor-critic with value distribution estimated by quantile regression. We evaluated this new algorithm, termed Distributional Advantage Actor-Critic (DA2C or QR-A2C) on a variety of tasks, and observed it to achieve at least as good as baseline algorithms, and outperforming baseline in some tasks with smaller variance and increased stability.

Motivation & Objective

• Address the challenge of training agents in visually complex, partially observable, and procedurally generated 3D navigation environments using raw RGB-D observations.
• Overcome the limitations of traditional RL algorithms like DQN and DDPG in sparse-reward, high-dimensional visual settings by integrating advanced techniques.
• Improve sample efficiency and learning stability through distributional RL and advantage-based value estimation in a deep reinforcement learning framework.
• Evaluate the proposed method on DeepMind Lab's 'SeekAvoid_Arena' and 'Maze Navigation' tasks to assess performance in complex, dynamic environments.
• Demonstrate the effectiveness of combining HER, actor-critic architecture, and distributional value learning for visual navigation without manual feature engineering.

Proposed method

• Adopt a distributional reinforcement learning framework to model the full return distribution rather than just the expected return, improving robustness to sparse rewards.
• Integrate advantage function estimation to refine value function learning, enhancing policy update stability and sample efficiency.
• Apply Hindsight Experience Replay (HER) to reframe failed trajectories by relabeling goals, providing meaningful learning signals in sparse-reward environments.
• Use a deep convolutional neural network to process raw RGB-D observations directly, eliminating the need for manual feature extraction.
• Implement an actor-critic architecture with separate networks for policy (actor) and value function (critic), leveraging off-policy training with experience replay.
• Extend the Q-function to depend on both state-action pairs and goals using Universal Value Function Approximators (UVFA), enabling goal-conditioned learning in dynamic environments.

Experimental results

Research questions

• RQ1 Can a distributional advantage actor-critic framework effectively learn navigation policies in visually complex, partially observable, and procedurally generated 3D environments using raw pixel inputs?
• RQ2 How does combining Hindsight Experience Replay (HER) with distributional and advantage-based value learning improve sample efficiency in sparse-reward navigation tasks?
• RQ3 To what extent does the integration of UVFA enable effective goal-conditioned learning in environments with changing start and goal states?
• RQ4 How does the proposed method compare to standard DQN and DDPG baselines in terms of learning speed and final performance on DeepMind Lab tasks?
• RQ5 Does end-to-end learning from raw pixels outperform linear models or manually engineered features in complex visual navigation scenarios?

Key findings

• The random policy achieved zero reward in 1000 steps across all environments, confirming the difficulty of the task and the need for structured learning approaches.
• The linear model with raw pixel inputs performed poorly, indicating that simple function approximators are insufficient for high-dimensional visual observations.
• Manual feature extraction was deemed non-generalizable, highlighting the need for end-to-end learning from raw pixels.
• The use of convolutional neural networks for feature extraction significantly improved performance over linear models, enabling effective representation learning from visual input.
• The integration of HER, distributional value learning, and advantage-based critic learning led to improved sample efficiency and faster convergence compared to standard DQN and DDPG.
• The proposed DAAC framework demonstrated superior performance on 'SeekAvoid_Arena' and 'Maze Navigation' tasks compared to baseline algorithms, particularly in handling sparse rewards and dynamic environments.