[Paper Review] Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Axel Abels, Diederik D.M. Roijers+3 more|arXiv (Cornell University)|Sep 20, 2018

Reinforcement Learning in Robotics41 citations

TL;DR

The paper introduces a Conditioned Network (CN) that outputs weight-conditioned multi-objective Q-values and a Diverse Experience Replay (DER) to handle non-stationarity in dynamic weights MORL, showing CN with DER outperforms adapted baselines on Minecart and DST under weight-change scenarios.

ABSTRACT

Many real-world decision problems are characterized by multiple conflicting objectives which must be balanced based on their relative importance. In the dynamic weights setting the relative importance changes over time and specialized algorithms that deal with such change, such as a tabular Reinforcement Learning (RL) algorithm by Natarajan and Tadepalli (2005), are required. However, this earlier work is not feasible for RL settings that necessitate the use of function approximators. We generalize across weight changes and high-dimensional inputs by proposing a multi-objective Q-network whose outputs are conditioned on the relative importance of objectives and we introduce Diverse Experience Replay (DER) to counter the inherent non-stationarity of the Dynamic Weights setting. We perform an extensive experimental evaluation and compare our methods to adapted algorithms from Deep Multi-Task/Multi-Objective Reinforcement Learning and show that our proposed network in combination with DER dominates these adapted algorithms across weight change scenarios and problem domains.

Motivation & Objective

• Address the dynamic weights setting where objective importances change over time.
• Develop a scalable deep MORL approach that generalizes across weight vectors and high-dimensional inputs.
• Improve sample efficiency and mitigate replay buffer bias in non-stationary weight scenarios.
• Demonstrate rapid adaptation to weight changes and convergence to optimal policies under varying regimes.

Proposed method

• Introduce a Conditioned Network (CN) that outputs multi-objective Q-value vectors conditioned on a weight vector.
• Train CN with an update rule that uses the active weight and a randomly sampled past weight per experience to balance current learning and policy retention.
• Propose Diverse Experience Replay (DER) to maintain trajectories that are informative for a broad range of weight vectors via diversity-based memorization and trajectory signatures.
• Use a Minecart benchmark with continuous state space, stochastic transitions, and delayed rewards to evaluate weight-change scenarios.
• Compare CN against UVFA, Multi-Network (MN), a baseline MO, and ablated CN variants (CN-ACTIVE, CN-UVFA) under sparse and regular weight-change regimes.
• Optionally evaluate standard experience replay (SER) with/without DER and incorporate prioritized replay.

Experimental results

Research questions

• RQ1 Can a weight-conditioned deep Q-network generalize across changing weight vectors in dynamic MORL?
• RQ2 Does Diverse Experience Replay improve sample efficiency and mitigate buffer bias across weight-change scenarios?
• RQ3 How does CN compare to adapted baselines (UVFA, MN, MO) in sparse versus regular weight-change settings?
• RQ4 Is CN robust to abrupt weight changes and capable of converging to optimal policies when weight changes are frequent?

Key findings

• CN dominates adapted algorithms across weight-change scenarios and problem domains.
• DER improves performance across all tested algorithms by maintaining diverse, informative trajectories in the replay buffer.
• CN rapidly adapts to sparse abrupt weight changes and converges to good policies under regular weight changes.
• MN requires long training times to converge to accurate Q-values, limiting its practical advantage in online dynamic weights.
• CN-ACTIVE often underperforms due to lack of mechanism to preserve past policies, while CN-UVFA without DER struggles due to learning on broad weight samples.
• Across Minecart and DST benchmarks, CN with DER achieves the best or near-best regret performance compared to baselines.