[Paper Review] Is Conditional Generative Modeling all you need for Decision-Making?

Anurag Ajay, Yilun Du+4 more|arXiv (Cornell University)|Nov 28, 2022

Language and cultural evolutionSocial Sciences31 citations

TL;DR

The paper reframes offline decision-making as conditional diffusion modeling, showing that a return-conditional diffusion model (Decision Diffuser) can yield competitive or superior policies without TD learning, and can flexibly handle constraints and skill composition.

ABSTRACT

Recent improvements in conditional generative modeling have made it possible to generate high-quality images from language descriptions alone. We investigate whether these methods can directly address the problem of sequential decision-making. We view decision-making not through the lens of reinforcement learning (RL), but rather through conditional generative modeling. To our surprise, we find that our formulation leads to policies that can outperform existing offline RL approaches across standard benchmarks. By modeling a policy as a return-conditional diffusion model, we illustrate how we may circumvent the need for dynamic programming and subsequently eliminate many of the complexities that come with traditional offline RL. We further demonstrate the advantages of modeling policies as conditional diffusion models by considering two other conditioning variables: constraints and skills. Conditioning on a single constraint or skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrate a composition of skills. Our results illustrate that conditional generative modeling is a powerful tool for decision-making.

Motivation & Objective

• Motivate using conditional generative models for sequential decision making beyond traditional RL.
• Show that a return-conditional diffusion model can stitch sub-optimal offline trajectories into high-return plans without value function estimation.
• Demonstrate conditioning on constraints and skills to produce composite behaviors at test time.
• Propose classifier-free guidance with low-temperature sampling to maximize trajectory returns from offline data.
• Provide evidence across standard benchmarks that conditional generative modeling can outperform several offline RL baselines.

Proposed method

• Model trajectories as state-only diffusion processes and use inverse dynamics to obtain actions.
• Train a reverse diffusion model p_theta to denoise noisy state sequences conditioned on y(tau) (returns, constraints, or skills).
• Use classifier-free guidance with low-temperature sampling to bias generation toward high-return or constraint-satisfying trajectories without explicit Q-functions.
• Condition on returns, constraints, or skills to generate behavior that maximally returns, satisfies multiple constraints, or composes skills.
• Incorporate an inverse dynamics model f_phi(s_t, s_{t+1}) to map generated state transitions to executable actions.
• Train jointly the diffusion model and inverse dynamics using maximum-likelihood style objectives and a denoising loss with occasional conditioning dropout.

Experimental results

Research questions

• RQ1 Can a return-conditioned diffusion model recover or surpass offline RL performance without dynamic programming or Q-function estimation?
• RQ2 Does conditioning on additional factors like constraints and skills enable flexible composition of behaviors at test time?
• RQ3 Is classifier-free guidance with low-temperature sampling effective for biasing offline data toward high-return trajectories?
• RQ4 How do diffusion-based policies compare to TD-based offline RL methods across standard benchmarks?
• RQ5 Can the approach handle multi-constraint and multi-skill scenarios and compose them during inference?

Key findings

• Decision Diffuser matches or outperforms several offline RL baselines (TD methods) on D4RL locomotion tasks and Kitchen tasks.
• Classifier-free guidance with low-temperature sampling improves trajectory quality and return-maximization compared to baseline diffusion models.
• Using inverse dynamics for action extraction yields better performance than diffusing over actions across evaluated environments.
• The method effectively satisfies single and multiple constraints in Kuka Block Stacking, outperforming BCQ and CQL which fail on some tasks.
• Skill composition experiments in Unitree-go-running show trajectories transitioning between gaits when conditioned on multiple skills; classifier-based analysis confirms gait switching in generated sequences.