[Paper Review] Off-Policy Evaluation for Action-Dependent Non-Stationary Environments

Yash Chandak, Shiv Shankar+4 more|arXiv (Cornell University)|Jan 24, 2023

Multi-Criteria Decision MakingDecision Sciences3 citations

TL;DR

This paper proposes OPEN, a novel off-policy evaluation method for non-stationary environments with action-dependent changes, using double counterfactual reasoning and importance-weighted instrument-variable regression to reduce bias and variance. It achieves superior performance in predicting future policy performance across active, passive, and hybrid non-stationary settings, outperforming prior methods even in passive cases.

ABSTRACT

Methods for sequential decision-making are often built upon a foundational assumption that the underlying decision process is stationary. This limits the application of such methods because real-world problems are often subject to changes due to external factors (passive non-stationarity), changes induced by interactions with the system itself (active non-stationarity), or both (hybrid non-stationarity). In this work, we take the first steps towards the fundamental challenge of on-policy and off-policy evaluation amidst structured changes due to active, passive, or hybrid non-stationarity. Towards this goal, we make a higher-order stationarity assumption such that non-stationarity results in changes over time, but the way changes happen is fixed. We propose, OPEN, an algorithm that uses a double application of counterfactual reasoning and a novel importance-weighted instrument-variable regression to obtain both a lower bias and a lower variance estimate of the structure in the changes of a policy's past performances. Finally, we show promising results on how OPEN can be used to predict future performances for several domains inspired by real-world applications that exhibit non-stationarity.

Motivation & Objective

• Address the fundamental challenge of off-policy evaluation in environments with active, passive, or hybrid non-stationarity, where traditional methods fail due to non-stationary dynamics.
• Overcome the limitations of existing off-policy evaluation methods that assume stationarity or rely on ergodicity/resetting, which are invalid in real-world long-horizon data.
• Develop a unified framework that infers structured changes in non-stationary environments without requiring full model estimation, focusing on policy performance prediction.
• Enable reliable future performance prediction of a target policy using only historical data from a behavior policy in non-stationary domains.
• Provide a model-free method that remains robust to hyperparameter choices and effectively handles both active and passive non-stationarity.

Proposed method

• Introduce a higher-order stationarity assumption, where non-stationarity evolves over time but the mechanism of change is fixed, enabling inference of structural change patterns.
• Apply double counterfactual reasoning: first to generate denoised estimates of past policy performance using importance sampling, then to forecast future performance via auto-regressive modeling.
• Use a two-stage instrument-variable regression to denoise importance sampling estimates, reducing variance in past performance estimation before forecasting.
• Leverage auto-regressive time-series modeling with lagged past performance to forecast future performance, making the method robust to model misspecification.
• Employ importance-weighted regression to correct for distributional shifts caused by action-dependent non-stationarity, ensuring consistent estimation across different policy behaviors.
• Design the method to be model-free and applicable to both episodic and continuing settings, avoiding assumptions of reset or steady-state distributions.

Figure 1: RoboToy domain.

Experimental results

Research questions

• RQ1 Can off-policy evaluation be reliably performed in environments with action-dependent non-stationarity, where past actions influence future system dynamics?
• RQ2 How can structured non-stationarity—whether passive, active, or hybrid—be leveraged to predict future policy performance without full environment model estimation?
• RQ3 Does a double application of counterfactual reasoning improve estimation accuracy in non-stationary settings, particularly in reducing bias and variance?
• RQ4 Can a method based on instrument-variable regression and auto-regressive forecasting outperform existing approaches designed for passive non-stationarity in both passive and active settings?
• RQ5 How robust is the proposed method to hyperparameter choices, especially when compared to parametric methods like Fourier-based regression?

Key findings

• OPEN significantly outperforms both WIS and Pro-WLS across all non-stationary domains, including those with active and hybrid non-stationarity, in terms of both bias and mean squared error (MSE).
• In passive non-stationarity settings, where no action-dependent changes occur, OPEN still provides accurate predictions despite the double counterfactual correction being theoretically redundant.
• Pro-WLS exhibits high bias and variance due to sensitivity to Fourier basis order, especially when the number of terms is insufficient or too large, leading to poor extrapolation.
• OPEN’s auto-regressive structure with a fixed lag of 300 terms provides greater robustness to hyperparameter choice than Pro-WLS’s parametric Fourier regression.
• The ablation study confirms that OPEN’s performance is stable across a wide range of lag values, indicating strong generalization and reliability in forecasting.
• With 30 independent trials per setting, OPEN consistently achieves lower MSE and bias than baselines across all domains and non-stationarity speeds, demonstrating statistical reliability.

Figure 2: Control Graph for the non-stationary process. See text for symbol definitions.