[Paper Review] Reinforcement Learning for Joint Optimization of Multiple Rewards

Mridul Agarwal, Vaneet Aggarwal|arXiv (Cornell University)|Sep 6, 2019

Age of Information Optimization57 references4 citations

TL;DR

This paper proposes model-based and model-free reinforcement learning algorithms for joint optimization of multiple rewards via non-linear functions of long-term average rewards, such as fairness metrics. It achieves a regret bound of $\tilde{O}\left(LKDS\sqrt{\frac{A}{T}}\right)$ and outperforms DQN and heuristic policies in fairness-aware scheduling tasks.

ABSTRACT

Finding optimal policies which maximize long term rewards of Markov Decision Processes requires the use of dynamic programming and backward induction to solve the Bellman optimality equation. However, many real-world problems require optimization of an objective that is non-linear in cumulative rewards for which dynamic programming cannot be applied directly. For example, in a resource allocation problem, one of the objectives is to maximize long-term fairness among the users. We notice that when an agent aim to optimize some function of the sum of rewards is considered, the problem loses its Markov nature. This paper addresses and formalizes the problem of optimizing a non-linear function of the long term average of rewards. We propose model-based and model-free algorithms to learn the policy, where the model-based policy is shown to achieve a regret of $\Tilde{O}\left(LKDS\sqrt{\frac{A}{T}} ight)$ for $K$ objectives combined with a concave $L$-Lipschitz function. Further, using the fairness in cellular base-station scheduling, and queueing system scheduling as examples, the proposed algorithm is shown to significantly outperform the conventional RL approaches.

Motivation & Objective

• To address the challenge of optimizing non-linear functions of cumulative rewards in Markov Decision Processes, where standard dynamic programming fails due to loss of Markov property.
• To formalize a novel average-per-step reward formulation for infinite-horizon sequential decision making under multiple objectives.
• To develop model-based and model-free RL algorithms that can learn stochastic policies for optimizing concave, Lipschitz-continuous joint objective functions.
• To achieve provable regret bounds in the absence of known transition probabilities, particularly for fairness objectives like proportional and α-fairness.
• To demonstrate superior performance over standard RL methods (e.g., DQN) and heuristics in real-world applications such as cellular base-station scheduling and multi-queue systems.

Proposed method

• Proposes a model-based algorithm using posterior sampling with Dirichlet distribution to estimate state transition probabilities and learn policies under uncertainty.
• Derives a regret bound of $\tilde{O}\left(LKDS\sqrt{\frac{A}{T}}\right)$ using Bellman error analysis for stochastic policies.
• Introduces a model-free policy gradient approach using neural networks to optimize stochastic policies without requiring transition model knowledge.
• Employs stochastic gradient ascent with adaptive learning rates and batch training to update policy parameters in continuous action spaces.
• Uses a Bellman error-based analysis to quantify deviation from optimal policy and link it to regret in time-horizon T.
• Applies the framework to fairness optimization by modeling objectives as concave, L-Lipschitz functions of average rewards, including α-fairness and weighted proportional fairness.

Experimental results

Research questions

• RQ1 Can reinforcement learning be effectively applied to optimize non-linear functions of cumulative rewards in MDPs where standard dynamic programming fails?
• RQ2 What regret bounds can be achieved by model-based and model-free RL algorithms when optimizing joint objectives like fairness in multi-agent systems?
• RQ3 How do the proposed algorithms compare to standard DQN and heuristic policies in fairness-critical scheduling tasks?
• RQ4 Can stochastic policies be effectively learned and optimized using policy gradient methods when the optimal policy is inherently non-deterministic?
• RQ5 Under what conditions does the joint optimization problem reduce to a convex or linear programming problem?

Key findings

• The model-based algorithm achieves a regret bound of $\tilde{O}\left(LKDS\sqrt{\frac{A}{T}}\right)$, which is sub-linear in time horizon T and scales favorably with number of objectives K and states S.
• The model-free policy gradient algorithm significantly outperforms DQN and LQF heuristics in maximizing α-fairness and weighted proportional fairness in queueing systems.
• In the α-fairness case, the proposed algorithm outperforms DQN and LQF, especially in steady-state performance, despite LQF being initially fairer in low-load conditions.
• For weighted proportional fairness with low arrival rates, the proposed policy gradient method consistently achieves higher fairness metrics than DQN across 50 runs, with median and inter-quartile range showing superior stability and performance.
• The framework enables effective optimization of max-min and concave joint objectives, with the problem reducing to linear programming under certain fairness metrics.
• The results demonstrate that standard Q-learning and DQN fail to capture non-linear reward functions, while the proposed methods succeed due to direct optimization of the joint objective.