[Paper Review] Interpretable Reinforcement Learning with Ensemble Methods

Alexander Brown, Marek Petrik|arXiv (Cornell University)|Sep 19, 2018

Reinforcement Learning in Robotics13 references3 citations

TL;DR

This paper proposes using gradient-boosted regression trees to create interpretable policies for reinforcement learning, training ensembles on policy data from neural networks or SARSA agents. The method achieves performance matching or exceeding the original agents in simple environments like Cart-Pole and Mountain Car, demonstrating that interpretable models can match high-performing RL systems while remaining human-readable.

ABSTRACT

We propose to use boosted regression trees as a way to compute human-interpretable solutions to reinforcement learning problems. Boosting combines several regression trees to improve their accuracy without significantly reducing their inherent interpretability. Prior work has focused independently on reinforcement learning and on interpretable machine learning, but there has been little progress in interpretable reinforcement learning. Our experimental results show that boosted regression trees compute solutions that are both interpretable and match the quality of leading reinforcement learning methods.

Motivation & Objective

• To address the lack of interpretable solutions in reinforcement learning, especially in high-stakes domains like healthcare and finance.
• To explore whether ensemble methods, particularly gradient-boosted decision trees, can produce policies that are both accurate and interpretable.
• To evaluate two approaches: (1) supervised learning on policy data from pre-trained agents, and (2) policy gradient boosting with decision trees from scratch.
• To investigate whether tree ensembles can resist overfitting and generalize better than the original RL agents.
• To assess the feasibility of space-efficient tree recycling in maintaining performance while limiting model size.

Proposed method

• Train a gradient-boosted ensemble of small regression trees to mimic the behavior of a pre-trained deep neural network policy in a reinforcement learning environment.
• Use supervised learning on state-action pairs collected from a trained RL agent (e.g., neural network or SARSA) to train the ensemble via gradient boosting.
• Apply policy gradient descent directly to decision trees by treating each tree as a policy parameter and updating via gradient ascent on expected return.
• Implement tree recycling by reusing older trees in the ensemble by training new trees on residuals and predictions from trees scheduled for removal.
• Use a value function approximation to reduce variance in policy gradient updates, similar to actor-critic methods.
• Limit tree depth and ensemble size to preserve interpretability, measuring interpretability via number of nodes or tree depth.

Experimental results

Research questions

• RQ1 Can a gradient-boosted ensemble of decision trees match the performance of a deep neural network policy in standard RL benchmarks?
• RQ2 Does the ensemble method improve generalization and resist overfitting compared to the original neural network policy?
• RQ3 Can policy gradient boosting with decision trees learn effective policies incrementally from scratch in simple MDPs?
• RQ4 Does tree recycling in the ensemble reduce model size without significantly degrading performance?
• RQ5 To what extent can the resulting policy be interpreted by humans, and how does interpretability trade-off with performance?

Key findings

• In the Cart-Pole environment, the ensemble trained on neural network policy data achieved perfect performance (200 timesteps per episode), outperforming the original neural network, which showed signs of overfitting.
• In the Mountain Car environment, the ensemble matched the performance of the SARSA agent, achieving the same episodic return, though not exceeding it.
• Policy gradient boosting with decision trees successfully increased cumulative reward over time, approaching the maximum of 200 per episode in Cart-Pole, but at a much slower rate than the neural network.
• The learning process via policy gradient boosting required 8,000–10,000 episodes to approach optimal performance, compared to ~400 episodes for the neural network.
• Tree recycling reduced performance significantly; after recycling began at episode 4,000, performance dipped and never recovered, indicating it is not a viable strategy for maintaining high reward.
• The ensemble method preserved interpretability by using small trees, and the resulting models were human-readable, with decision rules traceable through tree structures.