[Paper Review] Efficient Resource Allocation with Fairness Constraints in Restless Multi-Armed Bandits

De-Xun Li, Pradeep Varakantham|arXiv (Cornell University)|Jun 8, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper proposes a fairness-constrained approach for Restless Multi-Armed Bandits (RMAB) to ensure equitable intervention allocation across arms, particularly in public health and resource-limited settings. It introduces a novel fairness constraint based on maximum time since last activation, modifies the Whittle index algorithm for scalability and optimality under fairness, and evaluates both model-based and model-free learning methods, demonstrating that fairness can be enforced without significant loss in expected reward performance.

ABSTRACT

Restless Multi-Armed Bandits (RMAB) is an apt model to represent decision-making problems in public health interventions (e.g., tuberculosis, maternal, and child care), anti-poaching planning, sensor monitoring, personalized recommendations and many more. Existing research in RMAB has contributed mechanisms and theoretical results to a wide variety of settings, where the focus is on maximizing expected value. In this paper, we are interested in ensuring that RMAB decision making is also fair to different arms while maximizing expected value. In the context of public health settings, this would ensure that different people and/or communities are fairly represented while making public health intervention decisions. To achieve this goal, we formally define the fairness constraints in RMAB and provide planning and learning methods to solve RMAB in a fair manner. We demonstrate key theoretical properties of fair RMAB and experimentally demonstrate that our proposed methods handle fairness constraints without sacrificing significantly on solution quality.

Motivation & Objective

• To address the issue of intervention starvation in RMAB, where certain arms (e.g., disadvantaged communities) are consistently neglected due to reward-maximizing policies.
• To formalize a fairness constraint in RMAB that ensures every arm or type of arm is activated at least once within a bounded time window after its last activation.
• To develop a scalable, optimal Whittle index-based algorithm that respects fairness constraints in both finite and infinite horizon RMAB problems.
• To propose a model-free learning method (using Thompson sampling and Q-learning) for fair RMAB when transition probabilities are unknown.
• To empirically validate that fairness constraints can be enforced without significantly degrading expected reward performance.

Proposed method

• Proposes a fairness constraint defined by a maximum time interval $ L $, such that an arm not activated within $ L-1 $ steps must be selected at time $ L $, ensuring minimum activation frequency.
• Modifies the Whittle index algorithm to incorporate fairness constraints by introducing a penalty term in the index calculation that accounts for time since last activation.
• Derives a condition for optimal action selection under fairness constraints using a belief-state MDP, where the belief evolves based on passive transitions and action outcomes.
• Introduces a model-free learning approach using Thompson sampling and Q-learning to estimate Whittle indices when transition probabilities are unknown, enabling application in real-world settings.
• Uses a partially observable MDP framework where belief states are updated based on past actions and observations, with fairness-triggered activations modeled as transitions to the head of belief chains.
• Employs a value function approximation based on the work of Liu and Zhao (2010) to compute the expected future reward under fairness constraints.

Experimental results

Research questions

• RQ1 Can fairness constraints be formally defined in RMAB to prevent long-term neglect of certain arms or communities?
• RQ2 How can the Whittle index algorithm be adapted to maintain optimality and scalability while enforcing fairness constraints?
• RQ3 To what extent does enforcing fairness degrade the expected reward performance in RMAB settings?
• RQ4 Can model-free reinforcement learning methods effectively learn fair policies when transition dynamics are unknown?
• RQ5 How do different levels of fairness constraint strength (e.g., varying $ L $ and $ \eta $) affect the trade-off between fairness and reward?

Key findings

• The proposed fairness constraint ensures that no arm is starved for interventions, with up to 50% of arms receiving no activations under standard Whittle index, compared to consistent activation under the fair policy.
• The modified Whittle index algorithm maintains high solution quality, achieving average rewards close to the optimal non-fair baseline across all tested fairness constraint levels.
• For strong fairness constraints ($ L $ such that $ kL/N = 1.3 $), the method still achieves over 85% of the optimal reward, demonstrating minimal performance loss.
• The model-free learning approaches (Thompson sampling and Q-learning) successfully approximate the Whittle index and enforce fairness, with performance comparable to the model-based method.
• The method consistently outperforms random and myopic baselines across all intervention levels ($ k/N = 5\%, 10\%, 20\%, 30\% $), even under strict fairness constraints.
• The fairness constraint strength, controlled by $ L $ and $ \eta $, allows for tunable trade-offs between equity and efficiency, with low-strength constraints ($ kL/N = 3 $) resulting in minimal performance degradation.