[Paper Review] Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts

Xianwei Cao, Dou Quan+2 more|arXiv (Cornell University)|Mar 24, 2026

Decision-Making and Behavioral Economics0 citations

TL;DR

DPI maintains a probabilistic belief over changing objective weights and uses them to condition its policy, enabling online adaptation to context shifts in multi-objective RL. It outperforms fixed and heuristic baselines across dynamic queueing and maze tasks.

ABSTRACT

Humans often juggle multiple, sometimes conflicting objectives and shift their priorities as circumstances change, rather than following a fixed objective function. In contrast, most computational decision-making and multi-objective RL methods assume static preference weights or a known scalar reward. In this work, we study sequential decision-making problem when these preference weights are unobserved latent variables that drift with context. Specifically, we propose Dynamic Preference Inference (DPI), a cognitively inspired framework in which an agent maintains a probabilistic belief over preference weights, updates this belief from recent interaction, and conditions its policy on inferred preferences. We instantiate DPI as a variational preference inference module trained jointly with a preference-conditioned actor-critic, using vector-valued returns as evidence about latent trade-offs. In queueing, maze, and multi-objective continuous-control environments with event-driven changes in objectives, DPI adapts its inferred preferences to new regimes and achieves higher post-shift performance than fixed-weight and heuristic envelope baselines.

Motivation & Objective

• Motivate how humans adapt preferences under changing circumstances and translate this into a computational framework for dynamic value weighting.
• Formalize dynamic, unobserved preference weights as latent states that drift with context.
• Develop a variational inference module to estimate current preferences from history and a preference-conditioned policy to act on them.
• Demonstrate improved adaptation and interpretability in non-stationary environments like queueing and maze tasks.

Proposed method

• Introduce a two-module agent: a Value Appraisal module infers a distribution over latent preferences from recent history.
• Represent preferences as a latent z_t with omega_t = softmax(z_t) to model uncertainty and exploration in preference space.
• Train a preference-conditioned actor–critic with an on-policy envelope operator selecting among K sampled preferences.
• Optimize via an ELBO with a Boltzmann-rational likelihood of evidence and regularizers for stability (KL prior, direction alignment, self-consistency).
• Use vector-valued returns and vector GAE to compute per-dimension credit assignment, projecting to a scalar PPO objective for policy updates.

Experimental results

Research questions

• RQ1 Can online inference of dynamic, latent preference weights improve performance in non-stationary multi-objective tasks?
• RQ2 Do agents that infer and adapt preferences outperform fixed-weight or manually tuned baselines under context shifts?
• RQ3 Are the inferred preferences interpretable and aligned with task-relevant objectives?
• RQ4 Does the proposed DPI framework maintain stability and provide interpretable adaptation signals across different environments?

Key findings

• DPI achieves the highest mean episodic returns (MER) and speeds up post-shift recovery relative to baselines in Queue and Maze tasks.
• Post-shift performance shows DPI rapidly adapts after events, surpassing all baselines in short-horizon recovery.
• DPI yields a notable improvement in success rate, demonstrating robust task completion under dynamic contexts.
• In Maze, naive baselines fail to adapt; DPI maintains positive alignment between inferred preferences and the instantaneous reward structure.
• Ablations show removing KL regularization, directional alignment, or self-consistency degrades performance, highlighting the importance of each component.
• Interpretability analyses show DPI’s inferred preferences align semantically with task demands after events (e.g., deadlines, hazards, energy scarcity).