[Paper Review] Imitating Human Behaviour with Diffusion Models

Tim Pearce, Tabish Rashid+9 more|arXiv (Cornell University)|Jan 25, 2023

Human Motion and AnimationEngineering23 citations

TL;DR

Diffusion models are used to imitate human behavior in sequential environments by learning the full joint distribution over actions conditioned on observations, outperforming traditional behavior cloning baselines in robotic control and video game tasks. The paper also analyzes architectures, guidance, and sampling strategies for reliable sequential imitation.

ABSTRACT

Diffusion models have emerged as powerful generative models in the text-to-image domain. This paper studies their application as observation-to-action models for imitating human behaviour in sequential environments. Human behaviour is stochastic and multimodal, with structured correlations between action dimensions. Meanwhile, standard modelling choices in behaviour cloning are limited in their expressiveness and may introduce bias into the cloned policy. We begin by pointing out the limitations of these choices. We then propose that diffusion models are an excellent fit for imitating human behaviour, since they learn an expressive distribution over the joint action space. We introduce several innovations to make diffusion models suitable for sequential environments; designing suitable architectures, investigating the role of guidance, and developing reliable sampling strategies. Experimentally, diffusion models closely match human demonstrations in a simulated robotic control task and a modern 3D gaming environment.

Motivation & Objective

• Motivation to model the full, multimodal distribution of human actions conditioned on observations rather than relying on simplistic BC assumptions.
• Demonstrate that diffusion models can accurately capture complex action distributions in sequential environments.
• Develop architectural, guidance, and sampling innovations to adapt diffusion models to observation-to-action tasks.
• Evaluate diffusion-based BC on robotic control and modern video game environments to compare against strong baselines.

Proposed method

• Apply denoising diffusion probabilistic models (DDPM) to learn p(a|o) for observation-to-action tasks.
• Design architectures tailored for action vectors (Basic MLP, MLP Sieve, Transformer) with separate observation encoders and denoising networks.
• Investigate that Classifier-Free Guidance (CFG) harms performance in sequential observation-to-action tasks.
• Introduce reliable sampling schemes (Diffusion-X, Diffusion-KDE) to select high-likelihood actions during rollout.
• Compare diffusion-based BC to MSE, Discretised, K-Means, K-Means+Residual, and EBM baselines across tasks.

Experimental results

Research questions

• RQ1 Can diffusion models accurately model the full conditional distribution p(a|o) to imitate human demonstrations in sequential environments?
• RQ2 How do architectural choices (MLP, MLP Sieve, Transformer) affect diffusion-based BC performance?
• RQ3 What is the impact of guidance mechanisms (CFG) on sequential imitation?
• RQ4 Do reliable sampling schemes (Diffusion-X, Diffusion-KDE) improve robustness and fidelity of action sampling?
• RQ5 How do diffusion-based methods scale to complex environments (robotic control and CS:GO) compared to strong BC baselines?

Key findings

• Diffusion-based BC outperforms all baselines on robotic control tasks across multiple metrics, including Task completion rate and distributional similarity to humans.
• Architectural choices matter: Transformer and MLP Sieve outperform Basic MLP; Transformer yields the best metrics but with slower sampling.
• Classifier-Free Guidance (CFG) degrades performance in sequential observation-to-action tasks, increasing bias toward less common trajectories.
• Sampling schemes Diffusion-X and Diffusion-KDE improve reliability and fidelity over standard Diffusion BC, though KDE may reduce diversity.
• On a CS:GO-like video game task, Diffusion-X achieves best Wasserstein distance to human actions and competitive game scores, with slower but feasible sampling rates.