[Paper Review] Dyna Planning using a Feature Based Generative Model

Ryan Faulkner, Doina Precup|arXiv (Cornell University)|May 23, 2018

Reinforcement Learning in Robotics16 references3 citations

TL;DR

This paper proposes using deep belief networks (DBNs) as a non-linear generative model within the Dyna reinforcement learning framework to improve sample efficiency in high-dimensional state spaces, such as those with image observations. The approach significantly outperforms linear models in learning speed and performance, especially when generating high-quality simulated trajectories for value function learning.

ABSTRACT

Dyna-style reinforcement learning is a powerful approach for problems where not much real data is available. The main idea is to supplement real trajectories, or sequences of sampled states over time, with simulated ones sampled from a learned model of the environment. However, in large state spaces, the problem of learning a good generative model of the environment has been open so far. We propose to use deep belief networks to learn an environment model for use in Dyna. We present our approach and validate it empirically on problems where the state observations consist of images. Our results demonstrate that using deep belief networks, which are full generative models, significantly outperforms the use of linear expectation models, proposed in Sutton et al. (2008)

Motivation & Objective

• To address the challenge of learning effective generative models in large or continuous state spaces where traditional Dyna methods struggle.
• To improve sample efficiency in reinforcement learning by leveraging deep belief networks to model complex, high-dimensional observations like images.
• To demonstrate that non-linear generative models outperform linear expectation models in Dyna-style planning.
• To validate the approach in environments with image-based observations and non-deterministic transitions.

Proposed method

• The method uses deep belief networks (DBNs) to learn a hierarchical, non-linear representation of states from image observations.
• A temporal layer is trained between two DBNs to model the conditional distribution of the next state given the current state, per action.
• For each action, a separate DBN is trained to model state transitions, enabling action-conditioned generation of next states.
• The generative model produces simulated transitions by performing inference on the DBN using current state observations as input.
• These simulated transitions are used in conjunction with real experience to update the value function via TD(0) learning with linear function approximation.
• The learning rate for simulated samples decays over episodes, and epsilon-greedy exploration is used with annealing.

Experimental results

Research questions

• RQ1 Can deep belief networks effectively model complex, high-dimensional state transitions in reinforcement learning environments?
• RQ2 Does using a non-linear generative model in the Dyna framework lead to faster and more accurate value function learning than linear models?
• RQ3 How well do generative models perform in environments with image-based observations and stochastic dynamics?
• RQ4 To what extent does the quality of simulated trajectories impact the performance of Dyna planning?

Key findings

• The generative model trained with DBNs achieved high accuracy in predicting next states from image observations, enabling effective simulation.
• Dyna with the DBN-based generative model required fewer real environment interactions and less total data to achieve high returns compared to baseline TD(0)/SARSA.
• The DBN-based model produced higher-quality simulated trajectories than the linear model, allowing for longer and more effective planning.
• The linear model's simulated transitions degraded rapidly after a few steps (within ~10 steps), limiting its usefulness for long-horizon planning.
• Generative Dyna with K=50 simulated steps per real step achieved superior performance, with learning completed in about 14 hours for 7500 environment steps.
• The policy inferred from the generative Dyna model showed coherent, goal-directed behavior, as visualized in the grid-world walks.