[Paper Review] Unifying Large Language Model and Deep Reinforcement Learning for Human-in-Loop Interactive Socially-aware Navigation

Weizheng Wang, Obi, Ike+2 more|arXiv (Cornell University)|Mar 22, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes SRLM, a hybrid framework integrating Large Language Models (LLM) and Deep Reinforcement Learning (DRL) for human-in-loop interactive social robot navigation. By using an LLM for high-level command interpretation and a DRL-based planner for low-level control, enhanced by a Large Feedback Model (LFM) to stabilize action blending, SRLM achieves superior socially compliant navigation performance, outperforming baselines in both simulation and real-world tests with significant gains in social score and success rate.

ABSTRACT

Navigating human-filled spaces is crucial for the interactive social robots to support advanced services, such as cooperative carrying, which enables service provision in complex and crowded environments while adapting behavior based on real-time human language commands or feedback. However, existing social robot navigation planners face two major challenges: managing real-time user inputs and ensuring socially compliant behaviors in unfamiliar, zero-shot environments. In response, we introduce SALM, an interactive, human-in-loop Socially-Aware navigation Large Language Model framework that dynamically integrates deep reinforcement learning (DRL) with large language model (LLM) capabilities. SALM leverages contextual semantic understanding from real-time human-robot interactions to convert high-level user commands into precise, low-level control actions. A high-level LLM module parses user input, guiding the simultaneous generation of navigation commands by both a large language navigation model (LNM) and a DRL-based navigation model (RLNM). A memory mechanism archives temporal data for continuous refinement, while a multi-step graph-of-thoughts inference-based large language feedback model adaptively fuses the strengths of both planning approaches. Experimental evaluations demonstrate that SALM not only enhances navigational precision in crowded, dynamic environments but also significantly improves system adaptability, offering tailored behaviors that align with individual user preferences and real-time feedback. More details and videos about this work are available at: https://sites.google.com/view/navi-salm.

Motivation & Objective

• To address the challenge of real-time adaptation in social robot navigation under dynamic human feedback and volatile user preferences.
• To overcome the limitations of fixed-parameter DRL policies and LLMs' insensitivity to continuous control values in navigation tasks.
• To unify the strengths of LLMs (natural language understanding and long-context reasoning) with DRL (robust low-level control) for interactive, personalized navigation.
• To develop a feedback-driven system that aligns navigation policies with user preferences through human-in-the-loop reinforcement learning.
• To improve navigation robustness and social compliance in crowded, dynamic environments through a novel fusion mechanism between LLM and DRL planners.

Proposed method

• SRLM employs a Large Navigation Model (LNM) that encodes environmental and social information into textual embeddings for LLM-based global planning.
• A DRL-based planner (RLNM) provides benchmark-quality low-level motion control, ensuring stability in dynamic environments.
• A Large Feedback Model (LFM) dynamically blends actions from LNM and RLNM using chain-of-thought reasoning, adjusting fusion weights based on situational context.
• The system uses Reinforcement Learning from Human Feedback (RLHF) to align the reward function with user preferences, enabling personalization.
• A social navigation prompt and textual embedding encoder convert real-time sensor data into LLM-understandable input for action generation.
• The framework integrates LLM-driven high-level planning with DRL-driven execution, using LFM to mediate between the two for adaptive, socially compliant behavior.

Experimental results

Research questions

• RQ1 Can a hybrid LLM-DRL framework achieve real-time, interactive navigation in crowded human environments while adapting to dynamic user commands?
• RQ2 How effectively can an LLM-based planner generate low-level motion actions when combined with a DRL-based planner through a feedback mechanism?
• RQ3 To what extent does the Large Feedback Model (LFM) improve navigation stability and performance compared to fixed-weight fusion or standalone models?
• RQ4 Can the integration of LLM and DRL lead to better social compliance and success rates than state-of-the-art baselines in complex, dynamic scenarios?
• RQ5 How does the system maintain performance and adaptability after receiving real-time user feedback, especially when task objectives or preferences change?

Key findings

• SRLM achieved a social score (SS) of 90 and success rate (SR) of 90% in ablation studies, significantly outperforming SR-RLNM and other baselines.
• ANOVA results showed a highly significant F-value of 15410.139 (p < 0.0001), confirming that planner selection has a major impact on navigation performance.
• The LFM-based fusion mechanism enabled SRLM to maintain high performance after user feedback, while SR-RLNM failed to adapt to changed preferences.
• Trajectory visualization revealed that SR-LNM exhibited 'reciprocal dance' behavior—repetitive side-to-side motions—indicating instability in LLM-only control.
• SRLM demonstrated robustness in dynamic environments, with the LNM memory mechanism enabling long-term feedback evaluation and policy refinement.
• The integration of LLM and DRL via LFM led to a 90% success rate and 90 SS, confirming the effectiveness of the hybrid architecture in real-world and simulated settings.