[Paper Review] Surprising Negative Results for Generative Adversarial Tree Search

Kamyar Azizzadenesheli, Brandon Yang+3 more|arXiv (Cornell University)|Jun 15, 2018

Reinforcement Learning in Robotics47 references14 citations

TL;DR

This paper proposes Generative Adversarial Tree Search (GATS), a model-based deep reinforcement learning algorithm that uses a GAN-based dynamics model to generate environment rollouts and performs depth-limited Monte Carlo Tree Search (MCTS) with a DQN for value estimation. Surprisingly, despite theoretical advantages in bias-variance control and sample efficiency, GATS fails to outperform standard DQN in Atari environments, revealing critical limitations in depth-limited MCTS with learned generative models.

ABSTRACT

While many recent advances in deep reinforcement learning (RL) rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment model. In this work, we provide an extensive study on the design of deep generative models for RL environments and propose a sample efficient and robust method to learn the model of Atari environments. We deploy this model and propose generative adversarial tree search (GATS) a deep RL algorithm that learns the environment model and implements Monte Carlo tree search (MCTS) on the learned model for planning. While MCTS on the learned model is computationally expensive, similar to AlphaGo, GATS follows depth limited MCTS. GATS employs deep Q network (DQN) and learns a Q-function to assign values to the leaves of the tree in MCTS. We theoretical analyze GATS vis-a-vis the bias-variance trade-off and show GATS is able to mitigate the worst-case error in the Q-estimate. While we were expecting GATS to enjoy a better sample complexity and faster converges to better policies, surprisingly, GATS fails to outperform DQN. We provide a study on which we show why depth limited MCTS fails to perform desirably.

Motivation & Objective

• To design a sample-efficient, robust model-based RL algorithm using deep generative models for Atari environments.
• To investigate whether generative adversarial tree search (GATS) can outperform model-free DQN in sample efficiency and final policy performance.
• To analyze the bias-variance trade-off in Q-estimation within MCTS and assess its impact on planning quality.
• To evaluate the generalization and domain adaptation capabilities of the generative dynamics model (GDM) across different game modes and difficulties in ALE.
• To understand why depth-limited MCTS with a learned model fails to deliver expected performance gains in complex environments.

Proposed method

• GATS employs a generative dynamics model (GDM) based on conditional pix2pix GANs with Wasserstein loss and spectral normalization to generate high-fidelity successor frames from state-action pairs.
• The reward predictor (RP) classifies clipped rewards in [-1, 0, 1] for generated trajectories, enabling reward estimation in rollouts.
• A DQN or DDQN network is trained on real environment transitions to estimate Q-values for leaf nodes in MCTS, with a second Q-network fine-tuned on generated frames to align Q-values across real and generated states.
• GATS performs depth-limited MCTS using the GDM for rollouts and the Q-network for value assignment, with tree search guided by upper confidence bounds.
• The GDM is trained on real transitions and further fine-tuned using self-generated samples to extend rollout horizon beyond 10 steps while preserving accuracy.
• Domain adaptation is evaluated by transferring the GDM from one game mode/difficulty to another, with performance measured via L1/L2 loss and qualitative frame generation.

Experimental results

Research questions

• RQ1 Can a GAN-based generative dynamics model enable effective and sample-efficient planning in Atari environments via MCTS?
• RQ2 Does GATS achieve better sample complexity and faster convergence than standard DQN in model-based RL?
• RQ3 Why does depth-limited MCTS with a learned generative model fail to outperform model-free DQN despite theoretical advantages in bias-variance control?
• RQ4 How well does the GDM generalize across different game modes and difficulties in ALE, and what is its sample efficiency for domain adaptation?
• RQ5 To what extent do Q-value estimates on generated frames align with those on real frames, and how does this affect MCTS performance?

Key findings

• GATS fails to outperform standard DQN in any Atari environment tested, despite theoretical expectations of improved sample efficiency and reduced bias in Q-estimation.
• The GDM achieves low L1 and L2 losses on both training and test sets during domain adaptation, but qualitative analysis shows it fails to capture key dynamics like ball trajectory in Pong when trained from scratch.
• The GDM generalizes effectively to new game modes and difficulties, adapting with only thousands of samples—orders of magnitude fewer than required by the Q-network.
• The Q-network trained on generated frames (Qθ′) shows negligible difference in Q-value predictions compared to the real Q-network (Qθ), indicating good alignment between real and generated state representations.
• Rollout quality remains stable over 10+ steps when self-generated samples are used for continued training, preserving model fidelity during long-horizon planning.
• Despite robust GDM and RP performance, depth-limited MCTS with the learned model does not yield better policies, suggesting fundamental flaws in the planning process or value estimation in the tree search.