[Paper Review] MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning

Elise van der Pol, Daniel E. Worrall+3 more|UvA-DARE (University of Amsterdam)|Jun 30, 2020

Reinforcement Learning in Robotics44 references47 citations

TL;DR

Introduces MDP homomorphic networks that enforce group-structured equivariance in policy and value networks, enabling parameter sharing across symmetric state-action pairs to accelerate learning in DRL tasks.

ABSTRACT

This paper introduces MDP homomorphic networks for deep reinforcement learning. MDP homomorphic networks are neural networks that are equivariant under symmetries in the joint state-action space of an MDP. Current approaches to deep reinforcement learning do not usually exploit knowledge about such structure. By building this prior knowledge into policy and value networks using an equivariance constraint, we can reduce the size of the solution space. We specifically focus on group-structured symmetries (invertible transformations). Additionally, we introduce an easy method for constructing equivariant network layers numerically, so the system designer need not solve the constraints by hand, as is typically done. We construct MDP homomorphic MLPs and CNNs that are equivariant under either a group of reflections or rotations. We show that such networks converge faster than unstructured baselines on CartPole, a grid world and Pong.

Motivation & Objective

• Motivate and formalize exploiting symmetries in MDPs to reduce the solution space.
• Introduce neural networks that are equivariant under state-action symmetry groups.
• Provide a practical method to automatically construct equivariant network layers.
• Demonstrate faster convergence on standard RL benchmarks by exploiting symmetries.

Proposed method

• Define MDP homomorphisms and group-structured symmetries in the joint state-action space.
• Show that lifted policies are invariant and equivalently formulating the problem as group-equivariant networks.
• Propose a numerical procedure to construct equivariant layers using a symmetrizer that projects arbitrary weights into the equivariant subspace.
• Represent policy networks as linear combinations of basis equivariant weights learned from data.
• Apply the framework to MLPs, CNNs, and equivariant feature extractors across CartPole, grid world, and Pong.

Experimental results

Research questions

• RQ1 How can MDP symmetries be formalized as group-structured MDP homomorphisms to reduce the policy/value space?
• RQ2 Can neural networks be automatically constructed to be equivariant under these symmetries without hand-deriving constraints?
• RQ3 Do equivariant (MDP homomorphic) networks converge faster than non-equivariant baselines in symmetric RL tasks?
• RQ4 What is the impact of using equivariant feature extractors versus fully equivariant architectures on learning speed?

Key findings

• MDP homomorphic networks converge faster than unstructured baselines on CartPole, a grid world, and Pong.
• An automated method (symmetrizer) constructs equivariant layers without hand-deriving constraints.
• Equivariant basis networks outperform random or nullspace bases in convergence speed in several tasks.
• Equivariant feature extractors can yield faster convergence than full equivariant networks on some tasks.
• Data augmentation is less effective than embedding symmetries directly into network weights in Pong.
• Baseline CNNs in some cases show slower convergence compared to equivariant variants.