[Paper Review] A Theoretical Connection Between Statistical Physics and Reinforcement Learning

Jad Rahme, Ryan P. Adams|arXiv (Cornell University)|Jun 24, 2019

Reinforcement Learning in Robotics27 references4 citations

TL;DR

This paper establishes a theoretical connection between statistical physics and reinforcement learning by introducing a partition function $Χ(s,a,\beta)$ for Markov decision processes, which enables a novel dynamic programming approach without explicit value functions. The $\mathcal{Z}$-function obeys a linear Bellman equation in deterministic environments and yields Boltzmann-like policies that favor states with high trajectory diversity, offering a new model-free RL algorithm equivalent to expected SARSA with entropic regularization.

ABSTRACT

Sequential decision making in the presence of uncertainty and stochastic dynamics gives rise to distributions over state/action trajectories in reinforcement learning (RL) and optimal control problems. This observation has led to a variety of connections between RL and inference in probabilistic graphical models (PGMs). Here we explore a different dimension to this relationship, examining reinforcement learning using the tools and abstractions of statistical physics. The central object in the statistical physics abstraction is the idea of a partition function $\mathcal{Z}$, and here we construct a partition function from the ensemble of possible trajectories that an agent might take in a Markov decision process. Although value functions and $Q$-functions can be derived from this partition function and interpreted via average energies, the $\mathcal{Z}$-function provides an object with its own Bellman equation that can form the basis of alternative dynamic programming approaches. Moreover, when the MDP dynamics are deterministic, the Bellman equation for $\mathcal{Z}$ is linear, allowing direct solutions that are unavailable for the nonlinear equations associated with traditional value functions. The policies learned via these $\mathcal{Z}$-based Bellman updates are tightly linked to Boltzmann-like policy parameterizations. In addition to sampling actions proportionally to the exponential of the expected cumulative reward as Boltzmann policies would, these policies take entropy into account favoring states from which many outcomes are possible.

Motivation & Objective

• To establish a theoretical framework linking reinforcement learning and statistical physics using the partition function as a central abstraction.
• To derive a Bellman equation for the $\mathcal{Z}$-function that enables dynamic programming in deterministic MDPs.
• To extend the $\mathcal{Z}$-function formalism to stochastic environments via a variational approach.
• To develop a model-free RL algorithm based on $\mathcal{Z}$-function updates that avoids explicit value function representation.
• To show that policies derived from $\mathcal{Z}$ are Boltzmann-like and inherently favor states with high entropy and trajectory diversity.

Proposed method

• Construct a partition function $\mathcal{Z}(s,a,\beta)$ over state-action trajectories in an MDP, parameterized by inverse temperature $\beta$, analogous to statistical physics partition functions.
• Derive a Bellman equation for $\mathcal{Z}$, which becomes linear in deterministic environments, enabling direct solution methods.
• Define policies as $\pi(a|s) \propto \mathcal{Z}(s,a,\beta)$, yielding Boltzmann-like policies that incorporate entropy by favoring states with many possible outcomes.
• Extend the formalism to stochastic MDPs using a variational approximation to maintain consistency with the partition function framework.
• Propose a geometric update rule for $\mathcal{Z}$: $\mathcal{Z}(s_t,a_t,\beta) \leftarrow \mathcal{Z}(s_t,a_t,\beta)^{1-\alpha} \times \left(e^{\beta r_t + \mu} \sum_{a_{t+1}} \mathcal{Z}(s_{t+1},a_{t+1},\beta)\right)^\alpha$, replacing linear interpolation in Q-learning.
• Show that the induced $Q$-function update is equivalent to expected SARSA, enabling variance reduction and compatibility with standard exploration strategies.

Experimental results

Research questions

• RQ1 Can a partition function formalism from statistical physics be meaningfully applied to reinforcement learning problems?
• RQ2 Does the $\mathcal{Z}$-function in MDPs satisfy a Bellman equation, and is it solvable in closed form in deterministic settings?
• RQ3 How can the $\mathcal{Z}$-function be extended to stochastic environments while preserving physical and algorithmic consistency?
• RQ4 What are the policy properties induced by $\pi(a|s) \propto \mathcal{Z}(s,a,\beta)$, and how do they differ from standard Boltzmann policies?
• RQ5 Can a model-free RL algorithm be constructed using $\mathcal{Z}$-function updates that avoids explicit value function representation?

Key findings

• The $\mathcal{Z}$-function satisfies a Bellman equation that is linear in deterministic MDPs, enabling direct solution methods unavailable in standard value-function approaches.
• Value functions can be derived from the $\mathcal{Z}$-function as $Q(s,a,\beta) = \frac{\partial}{\partial\beta} \log \mathcal{Z}(s,a,\beta)$, interpreting them as average energies.
• Policies derived from $\mathcal{Z}$ are Boltzmann-like and favor states from which many trajectories are possible, incorporating entropy into exploration naturally.
• The proposed learning algorithm uses a geometric update rule for $\mathcal{Z}$, which induces an expected SARSA update for the $Q$-function, reducing variance compared to standard SARSA.
• The method enables model-free RL without explicit value function representation, offering a new paradigm grounded in statistical physics abstractions.
• The approach provides a natural entropic preference in policies, as $\mathcal{Z}$-based policies favor states with high trajectory diversity, a feature not explicitly captured in standard RL formulations.