[Paper Review] Hindsight Learning for MDPs with Exogenous Inputs

Sean R. Sinclair, Felipe Vieira Frujeri+7 more|arXiv (Cornell University)|Jul 13, 2022

Age of Information Optimization4 citations

TL;DR

This paper introduces Hindsight Learning (HL), a data-efficient reinforcement learning framework for Markov Decision Processes with Exogenous Inputs (Exo-MDPs), where uncertainty arises solely from external, uncontrollable variables. By leveraging past exogenous inputs to re-evaluate decisions in hindsight, HL improves policy learning with significantly reduced data requirements, outperforming both heuristic methods and state-of-the-art RL algorithms in multi-secretary, airline revenue management, and real-world cloud VM allocation tasks.

ABSTRACT

Many resource management problems require sequential decision-making under uncertainty, where the only uncertainty affecting the decision outcomes are exogenous variables outside the control of the decision-maker. We model these problems as Exo-MDPs (Markov Decision Processes with Exogenous Inputs) and design a class of data-efficient algorithms for them termed Hindsight Learning (HL). Our HL algorithms achieve data efficiency by leveraging a key insight: having samples of the exogenous variables, past decisions can be revisited in hindsight to infer counterfactual consequences that can accelerate policy improvements. We compare HL against classic baselines in the multi-secretary and airline revenue management problems. We also scale our algorithms to a business-critical cloud resource management problem -- allocating Virtual Machines (VMs) to physical machines, and simulate their performance with real datasets from a large public cloud provider. We find that HL algorithms outperform domain-specific heuristics, as well as state-of-the-art reinforcement learning methods.

Motivation & Objective

• Address the challenge of data inefficiency in reinforcement learning for resource management problems with uncontrollable external uncertainties.
• Model such problems as Exo-MDPs, where only exogenous inputs (e.g., VM requests, demand fluctuations) introduce uncertainty.
• Develop a novel learning framework that reuses historical exogenous inputs to generate counterfactual outcomes for past decisions, enabling more efficient policy improvement.
• Demonstrate superior performance of Hindsight Learning over domain-specific heuristics and state-of-the-art RL methods in both synthetic and real-world scenarios.
• Scale the method to a large-scale, business-critical cloud VM allocation problem using real production data, validating practical applicability and robustness.

Proposed method

• Formalize resource management problems as Exo-MDPs, where system dynamics depend on endogenous states and exogenous inputs that are independent of agent actions.
• Introduce the Hindsight Planner, which uses observed exogenous input sequences to compute counterfactual rewards for past decisions, enabling policy improvement without new environment interactions.
• Design a Hindsight Learning algorithm that imitates a Bayes-optimal selector by learning from these counterfactual outcomes, reducing variance and accelerating convergence.
• Apply value decomposition techniques to decompose the value function in Exo-MDPs, enabling scalable learning in high-dimensional state and input spaces.
• Use graph neural networks (GNNs) to represent VM and physical machine features, enabling the policy to generalize across complex allocation configurations.
• Implement a scalable hindsight heuristic for real-world VM allocation that prioritizes VMs by size and lifetime, achieving near-optimal performance with a dual gap of under 4% on average.

Experimental results

Research questions

• RQ1 Can hindsight re-evaluation of past decisions using observed exogenous inputs improve data efficiency in sequential decision-making under exogenous uncertainty?
• RQ2 How does Hindsight Learning compare to classic heuristics and state-of-the-art reinforcement learning methods in multi-secretary and airline revenue management problems?
• RQ3 To what extent can Hindsight Learning scale to real-world, large-scale cloud resource allocation with high-dimensional, continuous-time inputs?
• RQ4 How close is the proposed hindsight heuristic to the optimal solution in real-world VM allocation, and what is its empirical performance gap?
• RQ5 Can Hindsight Learning achieve better performance than end-to-end RL and heuristic baselines in production-scale cloud environments with real-world data?

Key findings

• Hindsight Learning achieved a performance of 0.18 ± 0.093 on the negative inverse packing density metric in the real-world VM allocation task, significantly outperforming DQN (-1.00 ± 0.41) and MAC (-0.38 ± 0.033).
• The Hindsight MAC algorithm reduced packing density by 0.05%, achieving a 1.5x improvement over the next best baseline, DQN, which only reduced it by 0.05%.
• In the stylized VM allocation environment, Hindsight MAC achieved a 0.05% packing density improvement, while the best baseline (MAC) achieved only -0.03%, indicating a clear data efficiency advantage.
• The hindsight heuristic achieved a dual gap of only 4.33% on average across 24 real-world clusters, with individual gaps ranging from 1.17% to 20.07%, indicating near-optimality in most cases.
• Hindsight Learning outperformed all baselines—including DQN, policy gradient, and AC—across all metrics in both synthetic and real-world settings, demonstrating consistent superiority.
• The method achieved a 1.5x improvement in packing density reduction compared to the next-best baseline, suggesting strong practical impact in reducing resource fragmentation and energy costs in cloud data centers.