[论文解读] Dynamic Weights in Multi-Objective Deep Reinforcement Learning

Axel Abels, Diederik D.M. Roijers+3 more|arXiv (Cornell University)|Sep 20, 2018

Reinforcement Learning in Robotics被引用 41

一句话总结

本文提出一个 Conditioned Network (CN)，输出基于权重条件的多目标 Q 值，以及一个 Diverse Experience Replay (DER)，用于处理动态权重 MORL 的非平稳性，并显示 CN 与 DER 在 Minecart 和 DST 的权重变化情景下优于改编基线方法。

ABSTRACT

Many real-world decision problems are characterized by multiple conflicting objectives which must be balanced based on their relative importance. In the dynamic weights setting the relative importance changes over time and specialized algorithms that deal with such change, such as a tabular Reinforcement Learning (RL) algorithm by Natarajan and Tadepalli (2005), are required. However, this earlier work is not feasible for RL settings that necessitate the use of function approximators. We generalize across weight changes and high-dimensional inputs by proposing a multi-objective Q-network whose outputs are conditioned on the relative importance of objectives and we introduce Diverse Experience Replay (DER) to counter the inherent non-stationarity of the Dynamic Weights setting. We perform an extensive experimental evaluation and compare our methods to adapted algorithms from Deep Multi-Task/Multi-Objective Reinforcement Learning and show that our proposed network in combination with DER dominates these adapted algorithms across weight change scenarios and problem domains.

研究动机与目标

• Address the dynamic weights setting where objective importances change over time.
• Develop a scalable deep MORL approach that generalizes across weight vectors and high-dimensional inputs.
• Improve sample efficiency and mitigate replay buffer bias in non-stationary weight scenarios.
• Demonstrate rapid adaptation to weight changes and convergence to optimal policies under varying regimes.

提出的方法

• Introduce a Conditioned Network (CN) that outputs multi-objective Q-value vectors conditioned on a weight vector.
• Train CN with an update rule that uses the active weight and a randomly sampled past weight per experience to balance current learning and policy retention.
• Propose Diverse Experience Replay (DER) to maintain trajectories that are informative for a broad range of weight vectors via diversity-based memorization and trajectory signatures.
• Use a Minecart benchmark with continuous state space, stochastic transitions, and delayed rewards to evaluate weight-change scenarios.
• Compare CN against UVFA, Multi-Network (MN), a baseline MO, and ablated CN variants (CN-ACTIVE, CN-UVFA) under sparse and regular weight-change regimes.
• Optionally evaluate standard experience replay (SER) with/without DER and incorporate prioritized replay.

实验结果

研究问题

• RQ1 Can a weight-conditioned deep Q-network generalize across changing weight vectors in dynamic MORL?
• RQ2 Does Diverse Experience Replay improve sample efficiency and mitigate buffer bias across weight-change scenarios?
• RQ3 How does CN compare to adapted baselines (UVFA, MN, MO) in sparse versus regular weight-change settings?
• RQ4 Is CN robust to abrupt weight changes and capable of converging to optimal policies when weight changes are frequent?

主要发现

• CN dominates adapted algorithms across weight-change scenarios and problem domains.
• DER improves performance across all tested algorithms by maintaining diverse, informative trajectories in the replay buffer.
• CN rapidly adapts to sparse abrupt weight changes and converges to good policies under regular weight changes.
• MN requires long training times to converge to accurate Q-values, limiting its practical advantage in online dynamic weights.
• CN-ACTIVE often underperforms due to lack of mechanism to preserve past policies, while CN-UVFA without DER struggles due to learning on broad weight samples.
• Across Minecart and DST benchmarks, CN with DER achieves the best or near-best regret performance compared to baselines.