[Paper Review] Navigation In Urban Environments Amongst Pedestrians Using Multi-Objective Deep Reinforcement Learning

Niranjan Deshpande, Dominique Vaufreydaz+1 more|arXiv (Cornell University)|Oct 11, 2021

Autonomous Vehicle Technology and Safety20 references10 citations

TL;DR

This paper proposes a multi-objective deep reinforcement learning approach using thresholded lexicographic Q-learning to improve autonomous navigation in urban environments with pedestrians. By training separate agents for safety and speed objectives and combining their policies, the method achieves 100% collision-free performance and reduced braking in both known and unknown environments, outperforming single-objective DQN baselines across all metrics.

ABSTRACT

Urban autonomous driving in the presence of pedestrians as vulnerable road users is still a challenging and less examined research problem. This work formulates navigation in urban environments as a multi objective reinforcement learning problem. A deep learning variant of thresholded lexicographic Q-learning is presented for autonomous navigation amongst pedestrians. The multi objective DQN agent is trained on a custom urban environment developed in CARLA simulator. The proposed method is evaluated by comparing it with a single objective DQN variant on known and unknown environments. Evaluation results show that the proposed method outperforms the single objective DQN variant with respect to all aspects.

Motivation & Objective

• To address the challenge of autonomous navigation in complex urban environments with pedestrians, where safety and speed are conflicting objectives.
• To overcome the limitations of single-objective reinforcement learning, which uses scalar rewards that poorly balance conflicting goals like collision avoidance and speed.
• To develop a multi-objective deep reinforcement learning framework that learns separate policies for safety and speed, then combines them into a unified decision-making policy.
• To evaluate the method in both known and unseen urban environments to assess robustness and generalization.
• To demonstrate that multi-objective learning leads to safer, smoother, and more efficient navigation than single-objective approaches.

Proposed method

• The method employs a multi-objective Markov Decision Process (MOMDP) with separate reward vectors for safety and speed objectives.
• A thresholded lexicographic Q-learning variant is used, where safety is prioritized over speed, ensuring collision avoidance is always met before optimizing for speed.
• Two agents are trained: one with a CNN for state processing and one with an LSTM to model temporal dependencies and pedestrian intent.
• The agents use separate Q-functions for each objective, with Q-values approximated via deep neural networks for scalability in high-dimensional state spaces.
• The final policy is derived by combining the outputs of both agents, ensuring safety is enforced first, followed by speed optimization.
• Training is conducted in a custom urban environment built in the CARLA simulator, featuring realistic pedestrian behaviors and traffic rules.

Experimental results

Research questions

• RQ1 Can a multi-objective deep reinforcement learning approach outperform single-objective DQN in autonomous navigation among pedestrians in urban settings?
• RQ2 How does the use of thresholded lexicographic Q-learning improve safety and driving smoothness compared to scalar reward-based methods?
• RQ3 To what extent does the inclusion of an LSTM-based agent enhance performance in terms of speed, stopping behavior, and pedestrian distance maintenance?
• RQ4 How does the proposed method generalize to unseen urban environments such as Town01 and Town04 in CARLA?
• RQ5 Does the multi-objective framework reduce unnecessary braking while maintaining collision-free navigation?

Key findings

• The multi-objective DQN agent achieved 100% collision-free episodes in both known and unknown environments, outperforming the single-objective DQN’s 97% and 98% in known and 95% and 97% in unknown environments.
• The multi-objective agent reduced average stops from 11.51 (single-objective) to 5.99 in unknown environments, indicating smoother, more natural driving behavior.
• In unknown environments, the multi-objective agent achieved a 98% success rate, compared to 95% for the single-objective agent, demonstrating better generalization.
• The multi-objective agent maintained an average distance of 0.75m to the closest pedestrian (within 2m), significantly higher than the single-objective agent’s 0.41m in unknown environments, indicating improved safety margins.
• The multi-objective agent achieved an average speed of 5.98 m/s in unknown environments, outperforming the single-objective agent’s 3.43 m/s, showing that safety and speed can be balanced effectively.
• The LSTM-based agent variant consistently outperformed the CNN-only variant in all metrics, particularly in reducing stops and improving crossing duration, highlighting the benefit of memory for intent prediction.