[Paper Review] Inverse Decision Modeling: Learning Interpretable Representations of Behavior

Daniel Jarrett, Alihan Hüyük+1 more|arXiv (Cornell University)|Oct 28, 2023

Explainable Artificial Intelligence (XAI)74 references4 citations

TL;DR

This paper introduces Inverse Decision Modeling (IDM), a unified framework for learning interpretable, parameterized representations of sequential decision behavior by formalizing both forward (normative) and inverse (descriptive) problems. It demonstrates how inverse bounded rational control enables learning of bounded rationality—capturing decision complexity, subjectivity, and uncertainty—through interpretable parameters like flexibility, optimism, and adaptivity.

ABSTRACT

Decision analysis deals with modeling and enhancing decision processes. A principal challenge in improving behavior is in obtaining a transparent description of existing behavior in the first place. In this paper, we develop an expressive, unifying perspective on inverse decision modeling: a framework for learning parameterized representations of sequential decision behavior. First, we formalize the forward problem (as a normative standard), subsuming common classes of control behavior. Second, we use this to formalize the inverse problem (as a descriptive model), generalizing existing work on imitation/reward learning -- while opening up a much broader class of research problems in behavior representation. Finally, we instantiate this approach with an example (inverse bounded rational control), illustrating how this structure enables learning (interpretable) representations of (bounded) rationality -- while naturally capturing intuitive notions of suboptimal actions, biased beliefs, and imperfect knowledge of environments.

Motivation & Objective

• To address the challenge of transparently characterizing imperfect, boundedly rational decision-making in real-world agents.
• To formalize a general inverse decision modeling framework that unifies and extends existing imitation and reward learning approaches.
• To enable interpretable representation of behavioral imperfections such as suboptimal actions, biased beliefs, and limited environmental knowledge.
• To develop a method that learns descriptive parameters (e.g., flexibility, optimism) from observed behavior, rather than assuming normative optimality.
• To provide a structured, parameterized model of bounded rationality that supports auditing and analysis of individual decision trajectories.

Proposed method

• Formalize the forward problem as a normative standard, subsuming optimal control and expected utility theory via a generalized planning operator.
• Define the inverse problem as a descriptive model that infers agent parameters (e.g., flexibility, optimism) from observed behavior using a dual optimization framework.
• Introduce inverse bounded rational control (IBRC) as a specific instantiation, modeling agents with bounded rationality through parameters like α⁻¹ (flexibility), β⁻¹ (optimism), and η⁻¹ (adaptivity).
• Use belief states (z) as sufficient statistics for subjective dynamics, enabling Markovianization of policy learning via belief-conditioned actions.
• Apply soft policy matching with KL divergence minimization to learn recognition and specification policies, reducing to a tractable optimization via belief propagation.
• Leverage Bellman operators and value function contraction to ensure stability and convergence of the inverse planning process.

Experimental results

Research questions

• RQ1 How can we formally represent and learn the parameters of bounded rationality from observed decision sequences?
• RQ2 In what ways does inverse decision modeling generalize existing imitation and reward learning frameworks?
• RQ3 How can we interpret behavioral imperfections—such as suboptimal actions or biased beliefs—as structured deviations from normative models?
• RQ4 What role do belief states and subjective dynamics play in enabling interpretable modeling of decision-making under uncertainty?
• RQ5 Can the framework support auditability of real-world decisions, such as clinical diagnostic trajectories?

Key findings

• The inverse bounded rational control (IBRC) framework successfully learns interpretable parameters—flexibility (α⁻¹), optimism (β⁻¹), and adaptivity (η⁻¹)—from observed behavior, enabling structured modeling of bounded rationality.
• The method achieves stable convergence via contraction of the Bellman operator, with ‖B V − B V′‖∞ ≤ γε, ensuring robustness in value function estimation.
• Belief trajectories in the belief simplex (e.g., for Alzheimer’s diagnosis) reveal interpretable decision patterns, such as delayed diagnosis due to bounded rationality.
• The framework captures intuitive behavioral phenomena: e.g., clinicians may avoid ordering MRIs due to high decision complexity or belief uncertainty, even when rational policy would recommend it.
• Soft policy matching via KL divergence minimization leads to tractable optimization, with entropy terms canceling out, resulting in Equation 24 for efficient training.
• The approach generalizes behavioral cloning and distribution matching, while enabling richer analysis beyond utility maximization—such as modeling optimism or risk-sensitivity.