[Paper Review] Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition

Justin Fu, Avi Singh+3 more|arXiv (Cornell University)|May 1, 2018

Neural dynamics and brain functionNeuroscience30 citations

TL;DR

This paper proposes variational inverse control with events, a data-driven framework that learns reward functions from goal states rather than full expert demonstrations by modeling goals as probabilistic events. It enables effective reinforcement learning in high-dimensional observation spaces like images, achieving strong performance without hand-specified rewards.

ABSTRACT

The design of a reward function often poses a major practical challenge to real-world applications of reinforcement learning. Approaches such as inverse reinforcement learning attempt to overcome this challenge, but require expert demonstrations, which can be difficult or expensive to obtain in practice. We propose inverse event-based control, which generalizes inverse reinforcement learning methods to cases where full demonstrations are not needed, such as when only samples of desired goal states are available. Our method is grounded in an alternative perspective on control and reinforcement learning, where an agent's goal is to maximize the probability that one or more events will happen at some point in the future, rather than maximizing cumulative rewards. We demonstrate the effectiveness of our methods on continuous control tasks, with a focus on high-dimensional observations like images where rewards are hard or even impossible to specify.

Motivation & Objective

• To address the challenge of defining reward functions in real-world reinforcement learning applications where expert demonstrations are costly or unavailable.
• To generalize inverse reinforcement learning by shifting focus from full trajectories to probabilistic event-based goals.
• To enable effective learning in high-dimensional observation spaces such as images, where rewards are difficult or impossible to specify manually.
• To develop a framework that maximizes the probability of desired future events rather than cumulative rewards.

Proposed method

• The method models an agent's goal as maximizing the probability that one or more events occur at some point in the future.
• It uses a variational inference framework to approximate the true inverse control policy from goal-state data.
• The approach formulates the reward function implicitly through event likelihoods, avoiding explicit reward shaping.
• It leverages deep neural networks to process high-dimensional observations like images and predict event probabilities.
• The framework is trained using a likelihood-based objective that encourages the agent to reach states where target events are likely.
• It generalizes inverse reinforcement learning by relaxing the need for full expert trajectories, instead using only positive goal-state samples.

Experimental results

Research questions

• RQ1 Can we learn effective policies without expert demonstrations by using only goal-state data?
• RQ2 How can we define reward functions in high-dimensional observation spaces such as images where manual specification is impractical?
• RQ3 Can event-based goal modeling outperform traditional inverse reinforcement learning when full trajectories are unavailable?
• RQ4 How does the variational inference framework enable robust inverse control under partial supervision?
• RQ5 What is the performance of event-based control in continuous control tasks with complex observations?

Key findings

• The method successfully learns policies in continuous control tasks using only goal-state samples, eliminating the need for expert trajectories.
• It achieves strong performance on tasks with high-dimensional observations, such as image-based environments, where traditional reward design fails.
• The event-based formulation enables effective learning even when only sparse goal data is available.
• The framework generalizes beyond standard inverse reinforcement learning by focusing on event likelihoods rather than full behavior cloning.
• Empirical results show that the approach matches or exceeds performance of baseline methods that require full demonstrations.