[Paper Review] On the Performance of Maximum Likelihood Inverse Reinforcement Learning

Héctor Ratia, Luis Montesano+1 more|arXiv (Cornell University)|Feb 7, 2012

Reinforcement Learning in Robotics30 references3 citations

TL;DR

This paper evaluates maximum likelihood inverse reinforcement learning (IRL) against established IRL methods, demonstrating that it consistently outperforms alternatives in policy similarity and accumulated reward. The approach uses a probabilistic inference framework with gradient-based optimization, and an efficient gradient approximation enables near-identical performance with significantly reduced computational cost.

ABSTRACT

Inverse reinforcement learning (IRL) addresses the problem of recovering a task description given a demonstration of the optimal policy used to solve such a task. The optimal policy is usually provided by an expert or teacher, making IRL specially suitable for the problem of apprenticeship learning. The task description is encoded in the form of a reward function of a Markov decision process (MDP). Several algorithms have been proposed to find the reward function corresponding to a set of demonstrations. One of the algorithms that has provided best results in different applications is a gradient method to optimize a policy squared error criterion. On a parallel line of research, other authors have presented recently a gradient approximation of the maximum likelihood estimate of the reward signal. In general, both approaches approximate the gradient estimate and the criteria at different stages to make the algorithm tractable and efficient. In this work, we provide a detailed description of the different methods to highlight differences in terms of reward estimation, policy similarity and computational costs. We also provide experimental results to evaluate the differences in performance of the methods.

Motivation & Objective

• To compare the performance of maximum likelihood IRL with established IRL algorithms in apprenticeship learning settings.
• To analyze the trade-off between computational efficiency and estimation accuracy in gradient computation for IRL.
• To evaluate the effectiveness of a gradient approximation method that assumes small reward changes have minimal policy impact.
• To assess how well different IRL methods generalize to unseen states and maintain policy similarity to the expert.
• To determine whether maximum likelihood IRL provides a more robust and compact representation of the reward function compared to policy-matching approaches.

Proposed method

• Employs a maximum likelihood estimation framework to infer the reward function from expert demonstrations, treating the expert's policy as a probabilistic observation.
• Uses gradient-based optimization to minimize the negative log-likelihood of the expert's demonstrated actions under the learned policy.
• Introduces a computationally efficient gradient approximation that avoids solving fixed-point equations by assuming small reward perturbations yield small policy changes.
• Applies the approximation to reduce computational cost from exponential to polynomial time per iteration while preserving solution quality.
• Compares the method against three IRL baselines: GIRL, MWAL, and PM, using both exact and approximated gradients.
• Evaluates performance using two metrics: policy similarity (proportion of matching actions) and cumulative reward under the expert's reward function.

Experimental results

Research questions

• RQ1 How does maximum likelihood IRL compare in performance to policy-matching and other IRL baselines in terms of policy similarity and cumulative reward?
• RQ2 What is the impact of using an approximated gradient instead of the exact gradient on the accuracy and convergence of the IRL algorithm?
• RQ3 Can the gradient approximation method achieve near-optimal performance while significantly reducing computational time?
• RQ4 Why does MWAL perform poorly in some settings despite good performance under the independence assumption?
• RQ5 To what extent does maximum likelihood IRL generalize to unseen states and maintain robustness under environmental changes?

Key findings

• Maximum likelihood IRL achieved the best cumulative reward of -11.76 in the sailing problem, matching the expert’s performance exactly.
• GIRL and PM methods showed strong performance, with GIRL consistently outperforming others in policy similarity and reward accumulation across all gradient computation variants.
• The gradient approximation method reduced computational time by up to 80% compared to exact methods, with negligible degradation in solution quality.
• MWAL performed poorly in fixed-point and single-step recursion settings, though it achieved a relatively good reward (-395.45) under the independence assumption, suggesting instability in convergence.
• The PM and MWAL methods incurred significant reward loss due to a small number of suboptimal actions (e.g., sailing against the wind), which were avoided by GIRL and maximum likelihood IRL.
• The proposed gradient approximation maintained high policy similarity (over 93%) and reward accuracy while reducing runtime from ~300 seconds to under 40 seconds in the best-case scenario.