[Paper Review] Dynamic value alignment through preference aggregation of multiple objectives

Marcin Korecki, Damian Dailisan+1 more|arXiv (Cornell University)|Oct 9, 2023

Autonomous Vehicle Technology and SafetyEngineering3 citations

TL;DR

This paper proposes a dynamic value alignment method for reinforcement learning agents using preference aggregation across multiple objectives, enabling the agent to adapt to shifting human preferences in real time. By training separate Deep Q-Networks for each objective and using proportional voting to aggregate preferences, the approach avoids reward hacking and achieves balanced performance across speed, stops, and wait times—outperforming single-objective methods that prioritize one metric at the cost of fairness.

ABSTRACT

The development of ethical AI systems is currently geared toward setting objective functions that align with human objectives. However, finding such functions remains a research challenge, while in RL, setting rewards by hand is a fairly standard approach. We present a methodology for dynamic value alignment, where the values that are to be aligned with are dynamically changing, using a multiple-objective approach. We apply this approach to extend Deep $Q$-Learning to accommodate multiple objectives and evaluate this method on a simplified two-leg intersection controlled by a switching agent.Our approach dynamically accommodates the preferences of drivers on the system and achieves better overall performance across three metrics (speeds, stops, and waits) while integrating objectives that have competing or conflicting actions.

Motivation & Objective

• To address the challenge of static reward functions in RL that can lead to misaligned, unfair policies due to fixed optimization goals.
• To enable AI agents to dynamically adapt to shifting human preferences in real-world control systems such as traffic intersections.
• To improve fairness and system-wide performance by aggregating multiple conflicting objectives (e.g., speed, stops, wait times) through participatory voting.
• To demonstrate that multi-objective aggregation via voting outperforms single-objective reward shaping, especially in avoiding suboptimal or inequitable solutions.
• To explore scalable, decentralized methods for value alignment that avoid the pitfalls of fixed reward engineering and algorithmic dictatorship.

Proposed method

• The method extends Deep Q-Learning by training separate Deep Q-Networks (DQNs) for each objective (e.g., minimizing stops, minimizing wait times).
• At each decision step, the system collects preferences from simulated drivers (representing different user groups) via proportional voting on which objective should take precedence.
• The final action is selected based on a weighted aggregation of Q-values from all objectives, where weights are determined by the proportion of votes for each objective.
• The voting mechanism uses proportional representation to reduce issues like 'tyranny of the majority' and 'wasted votes' common in simple majority voting.
• The approach avoids the need to pre-specify fixed reward weights, which can lead to misalignment if incorrectly set.
• The method is evaluated in a simulated two-leg intersection environment where agents control traffic light signals based on dynamic user preferences.

Figure 1: Two-way intersection controlled by an RL agent. The agent controls which direction has a green signal at any given time.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent dynamically align with shifting human preferences in a multi-objective control system?
• RQ2 How does preference aggregation via proportional voting compare to majority voting in terms of fairness and system performance?
• RQ3 Can multi-objective aggregation prevent reward hacking and suboptimal policies that emerge from single-objective reward shaping?
• RQ4 What is the impact of dynamically adjusting objective weights based on real-time user input on key performance metrics like speed, stops, and wait times?
• RQ5 Does the method scale effectively to multiple objectives without requiring retraining or manual weight tuning?

Key findings

• The proposed method successfully avoids the 'reward-hacked' solution seen in single-objective training, where minimizing stops led to excessive waiting times for one direction.
• By aggregating preferences through proportional voting, the system achieved balanced performance across all three metrics—speed, stops, and wait times—without favoring any single objective.
• The method outperformed a linear combination of objectives with fixed weights, which led to a suboptimal equilibrium with near-zero stops but extremely high wait times.
• Proportional voting was found to be more effective than majority voting in mitigating fairness issues such as 'tyranny of the majority' and ensuring representation of minority preferences.
• The approach demonstrated robustness to varying preference distributions, even with a 50-50 split in user preferences, and maintained stable performance across simulations.
• The method provides a scalable alternative to traditional multi-objective RL, avoiding the need for retraining when new objectives are added or weights are adjusted.

Figure 2: Multi-objective action selection uses multiple Deep $Q$ -Networks (DQN) each trained purely on a single objective. For a given environment state $s$ , each DQN outputs a vector of $Q$ -values. A softmax re-scales these values to reflect their relative importance for a given DQN. These are