[Paper Review] Entropy Regularized Motion Planning via Stein Variational Inference

Alexander Lambert, Byron Boots|arXiv (Cornell University)|Jul 11, 2021

Reinforcement Learning in Robotics20 references4 citations

TL;DR

This paper proposes a particle-based motion planning algorithm that uses Stein Variational Inference with entropy regularization to generate diverse, low-cost, and collision-free trajectories for high-degree-of-freedom systems. By formulating trajectory optimization as probabilistic inference and leveraging a Gauss-Newton Hessian approximation, the method efficiently samples from multi-modal posterior distributions over trajectories, enabling robust planning and value function estimation for imitation and reinforcement learning.

ABSTRACT

Many Imitation and Reinforcement Learning approaches rely on the availability of expert-generated demonstrations for learning policies or value functions from data. Obtaining a reliable distribution of trajectories from motion planners is non-trivial, since it must broadly cover the space of states likely to be encountered during execution while also satisfying task-based constraints. We propose a sampling strategy based on variational inference to generate distributions of feasible, low-cost trajectories for high-dof motion planning tasks. This includes a distributed, particle-based motion planning algorithm which leverages a structured graphical representations for inference over multi-modal posterior distributions. We also make explicit connections to both approximate inference for trajectory optimization and entropy-regularized reinforcement learning.

Motivation & Objective

• To generate a diverse, low-cost distribution of feasible trajectories for high-degree-of-freedom motion planning tasks.
• To address the challenge of multi-modality in trajectory posteriors under task constraints and obstacle avoidance.
• To enable sample-efficient imitation and reinforcement learning by providing a rich distribution of expert-like trajectories.
• To improve upon existing trajectory optimization methods by incorporating uncertainty and entropy regularization through variational inference.
• To develop a scalable, particle-based inference framework that leverages structured graphical models and curvature information for efficient sampling.

Proposed method

• Formulates motion planning as Bayesian posterior inference over trajectories, where optimality is encoded via a binary indicator variable and cost functions are modeled as likelihoods.
• Uses a Gaussian Process prior over trajectory parameters to enforce smoothness and enables efficient interpolation from sparse support states.
• Applies Stein Variational Gradient Descent (SVG-D) with anisotropic kernel and Hessian-based metric to iteratively update particle positions in a way that minimizes KL divergence to the target posterior.
• Employs a sparse Gauss-Newton approximation of the Hessian to reduce computational cost while preserving the sparsity and structure of the problem.
• Derives a particle-based value function estimate using the empirical posterior distribution, enabling use in model-based RL and imitation learning.
• Integrates curvature-aware updates via the average Hessian across particles, improving convergence and exploration in high-dimensional trajectory spaces.

Experimental results

Research questions

• RQ1 How can we efficiently sample from a multi-modal posterior distribution of low-cost, collision-free trajectories in high-dimensional motion planning?
• RQ2 What is the role of entropy regularization in improving the diversity and robustness of trajectory distributions for imitation learning?
• RQ3 Can Stein variational inference with Hessian-based metrics outperform standard trajectory optimization in terms of convergence and distribution quality?
• RQ4 How can we scale probabilistic motion planning to high-degree-of-freedom systems using structured graphical models and sparse linear algebra?
• RQ5 To what extent can particle-based posterior approximation improve value function estimation for model-based reinforcement learning?

Key findings

• The proposed method successfully generates diverse, low-cost, and collision-free trajectories in a 7-dof manipulation task, as demonstrated in visualizations and a supplementary video.
• The particle-based posterior approximation enables accurate estimation of the optimal value function, with Eq. (25) providing a differentiable and scalable estimate.
• Using the average Hessian as a preconditioner improves convergence and enables efficient, curvature-aware updates in high-dimensional trajectory spaces.
• The algorithm recovers GPMP2 in the single-particle limit, validating its consistency with established methods.
• The method achieves effective exploration of multi-modal trajectory distributions even within a single homotopy class, as shown in qualitative results.
• The use of sparse linear solvers and structured Hessian approximations enables significant computational speed-ups, especially for long-horizon planning.