[Paper Review] Internal Model from Observations for Reward Shaping

Daiki Kimura, Subhajit Chaudhury+2 more|arXiv (Cornell University)|Jun 2, 2018

Reinforcement Learning in Robotics28 references13 citations

TL;DR

This paper proposes a reinforcement learning method that learns an internal predictive model from expert-only state trajectories (e.g., game videos) to generate dense, intrinsic rewards without requiring action labels. By measuring the discrepancy between actual and predicted states, the agent shapes rewards to encourage behavior close to expert demonstrations, achieving faster and more stable learning than sparse or hand-crafted rewards in environments like Super Mario Bros and Flappy Bird.

ABSTRACT

Reinforcement learning methods require careful design involving a reward function to obtain the desired action policy for a given task. In the absence of hand-crafted reward functions, prior work on the topic has proposed several methods for reward estimation by using expert state trajectories and action pairs. However, there are cases where complete or good action information cannot be obtained from expert demonstrations. We propose a novel reinforcement learning method in which the agent learns an internal model of observation on the basis of expert-demonstrated state trajectories to estimate rewards without completely learning the dynamics of the external environment from state-action pairs. The internal model is obtained in the form of a predictive model for the given expert state distribution. During reinforcement learning, the agent predicts the reward as a function of the difference between the actual state and the state predicted by the internal model. We conducted multiple experiments in environments of varying complexity, including the Super Mario Bros and Flappy Bird games. We show our method successfully trains good policies directly from expert game-play videos.

Motivation & Objective

• To address the challenge of sparse or unavailable reward functions in reinforcement learning.
• To enable imitation learning from expert state trajectories alone, without access to action sequences.
• To develop a reward shaping method that uses internal predictive modeling to guide policy learning.
• To demonstrate that dense intrinsic rewards can be derived from video demonstrations alone, enabling effective RL training.

Proposed method

• The agent learns a 3D-CNN-based internal model that predicts the next frame in a sequence of states from expert gameplay videos.
• The internal model is trained solely on state trajectories, without any action or reward information.
• During RL, the agent computes a reward as the negative L2 distance between the actual next state and the predicted state from the internal model.
• A thresholding mechanism is applied to prevent trivial solutions, such as staying still at the start state.
• The reward function is designed to be dense and intrinsic, encouraging the agent to follow expert-like state transitions.
• The method is evaluated using DQN, DDPG, and PPO in environments like Super Mario Bros and Flappy Bird with only video demonstrations.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent learn a good policy using only expert state trajectories from game videos, without any action or reward signals?
• RQ2 How does the performance of a reward shaping method based on internal model prediction compare to hand-crafted dense rewards and curiosity-based methods?
• RQ3 Can the internal model generate a dense, meaningful reward signal that enables faster and more stable learning than sparse rewards?
• RQ4 To what extent can the internal model generalize across different environments with varying task complexity?
• RQ5 How sensitive is the method to hyperparameters like the threshold ζ, and what impact does model architecture have on performance?

Key findings

• The proposed method achieved faster learning convergence than both hand-crafted dense rewards and curiosity-based methods in the Super Mario Bros environment.
• Even with sparse rewards, the agent failed to reach the goal consistently (average position 650) after 3 million steps, whereas the proposed method reached better performance more reliably.
• The internal model-based reward outperformed curiosity-based and score-based rewards in learning speed and final policy quality.
• The method successfully trained agents using only video demonstrations, without requiring action annotations or reward signals.
• The use of a threshold ζ was critical to prevent trivial policies, such as remaining stationary at the initial state.
• Performance improved with deeper networks and higher-resolution inputs, suggesting room for further optimization.