[Paper Review] Multi-Objective Deep Q-Learning with Subsumption Architecture

Tomasz Tajmajer|arXiv (Cornell University)|Apr 21, 2017

Visual Attention and Saliency Detection7 citations

TL;DR

This paper proposes a multi-objective deep reinforcement learning framework using separate Deep Q-Networks (DQNs) for individual objectives, coordinated via signal suppression inspired by Brooks' subsumption architecture. The method enables modular, replaceable behavior modules that collectively produce a single action, achieving performance comparable to monolithic DQNs while supporting decoupled learning and dynamic objective prioritization in a 2D visual environment.

ABSTRACT

In this work we present a method for using Deep Q-Networks (DQNs) in multi-objective tasks. Deep Q-Networks provide remarkable performance in single objective tasks learning from high-level visual perception. However, in many scenarios (e.g in robotics), the agent needs to pursue multiple objectives simultaneously. We propose an architecture in which separate DQNs are used to control the agent's behaviour with respect to particular objectives. In this architecture we use signal suppression, known from the (Brooks) subsumption architecture, to combine outputs of several DQNs into a single action. Our architecture enables the decomposition of the agent's behaviour into controllable and replaceable sub-behaviours learned by distinct modules. To evaluate our solution we used a game-like simulator in which an agent - provided with high-level visual input - pursues multiple objectives in a 2D world. Our solution provides benefits of modularity, while its performance is comparable to the monolithic approach.

Motivation & Objective

• To address the challenge of training agents to pursue multiple objectives simultaneously in complex, visually rich environments.
• To enable modular behavior design by decomposing multi-objective tasks into independent, learnable DQN modules.
• To maintain high performance comparable to monolithic DQN approaches while improving maintainability and adaptability.
• To integrate signal suppression from the subsumption architecture to resolve conflicts between concurrent objectives.

Proposed method

• The architecture employs multiple independent DQNs, each trained to optimize a specific objective.
• Each DQN processes high-level visual input and outputs an action vector for its respective objective.
• Signal suppression is applied to prioritize higher-priority DQN outputs, effectively subsuming lower-priority actions.
• The final action is selected based on a priority-based merging mechanism that suppresses lower-level actions when higher-level actions are active.
• The system supports dynamic reconfiguration of objectives by enabling or disabling individual DQN modules.
• The approach is evaluated in a 2D game-like simulator with continuous visual observations and multiple simultaneous goals.

Experimental results

Research questions

• RQ1 Can a modular DQN architecture with signal suppression effectively manage multiple objectives in a visual reinforcement learning setting?
• RQ2 How does the performance of the proposed multi-objective DQN compare to a monolithic DQN in the same environment?
• RQ3 To what extent can individual behavior modules be replaced or reconfigured without disrupting overall agent performance?
• RQ4 How well does signal suppression from the subsumption architecture coordinate conflicting actions from multiple DQNs?
• RQ5 Does modularity improve the interpretability and maintainability of multi-objective policies without sacrificing learning efficiency?

Key findings

• The proposed architecture achieves performance comparable to a monolithic DQN in the multi-objective 2D simulator.
• Modular behavior design enables independent training and replacement of individual objectives without retraining the entire system.
• Signal suppression effectively resolves action conflicts by prioritizing higher-level objectives, mimicking hierarchical control.
• The system demonstrates robustness to changes in objective priorities through dynamic module activation.
• The decomposition into separate DQNs improves the interpretability of learned behaviors while maintaining competitive learning efficiency.