[Paper Review] Learning Object-conditioned Exploration using Distributed Soft Actor Critic

Ayzaan Wahid, Austin V. Stone+3 more|arXiv (Cornell University)|Jul 28, 2020

Reinforcement Learning in Robotics11 citations

TL;DR

This paper proposes a scalable, distributed Soft Actor-Critic (SAC) reinforcement learning framework for end-to-end object-conditioned exploration and low-level control in simulated robotic navigation. Trained on 98 million experience steps in 24 hours using 8 GPUs, the method achieves a 68% success rate and 0.58 SPL on unseen, visually complex home environments, learning natural exploration behaviors and continuous control without explicit supervision.

ABSTRACT

Object navigation is defined as navigating to an object of a given label in a complex, unexplored environment. In its general form, this problem poses several challenges for Robotics: semantic exploration of unknown environments in search of an object and low-level control. In this work we study object-guided exploration and low-level control, and present an end-to-end trained navigation policy achieving a success rate of 0.68 and SPL of 0.58 on unseen, visually complex scans of real homes. We propose a highly scalable implementation of an off-policy Reinforcement Learning algorithm, distributed Soft Actor Critic, which allows the system to utilize 98M experience steps in 24 hours on 8 GPUs. Our system learns to control a differential drive mobile base in simulation from a stack of high dimensional observations commonly used on robotic platforms. The learned policy is capable of object-guided exploratory behaviors and low-level control learned from pure experiences in realistic environments.

Motivation & Objective

• To develop a scalable, end-to-end reinforcement learning policy for object navigation in visually complex, unstructured environments with no prior knowledge of the layout or object location.
• To address the dual challenges of semantic exploration (finding objects via contextual cues) and low-level continuous control (manipulating a differential drive robot) in realistic simulation.
• To demonstrate that emergent exploration behaviors—such as peering into rooms, backtracking, and beelining to visible targets—can arise from pure experience without explicit supervision.
• To evaluate the impact of sensor modalities (e.g., RGBD, LiDAR) on navigation performance and robustness, especially in narrow passages and long-distance navigation.

Proposed method

• The method employs a distributed implementation of the Soft Actor-Critic (SAC) algorithm, enabling parallel experience collection and model optimization across multiple GPUs.
• A ResNet50 + LSTM architecture processes high-dimensional observations (e.g., RGB, depth, LiDAR) to output continuous control commands for a differential drive robot.
• The system is trained end-to-end in a simulation environment based on real home scans, with physics modeling for realistic low-level control.
• Experience is collected from interactions in visually rich, complex environments, with no prior map or localization information.
• The policy is trained using off-policy data, maximizing sample efficiency to reduce data collection cost.
• Sensors include RGB, depth, and LiDAR, with ablation studies on the impact of object detection (Det) on performance.

Experimental results

Research questions

• RQ1 Can a single end-to-end policy learn to perform object-conditioned exploration and low-level control in visually complex, real-world-like simulated environments without explicit supervision?
• RQ2 How does the inclusion of LiDAR or object detection (Det) affect navigation performance, especially in long-range or narrow-path scenarios?
• RQ3 What emergent exploration behaviors—such as peering, circling, or backtracking—arise naturally from pure RL training in this setup?
• RQ4 To what extent do memory limitations (e.g., LSTM) affect long-term navigation, and what failure modes emerge as a result?
• RQ5 How scalable is the distributed SAC framework in terms of experience collection and training efficiency?

Key findings

• The system achieves a success rate of 68% and a SPL of 0.58 on unseen, visually complex scans of real homes, demonstrating strong generalization.
• The policy learns natural, context-aware exploration behaviors such as peeking into rooms, circling to gather 360° views, and backtracking when the object is not found.
• Beeline navigation to the target emerges only after the object is visually detected, indicating effective integration of perception and control.
• The use of LiDAR significantly improves performance on long-distance navigation tasks, particularly in narrow doorways, due to better collision avoidance.
• Object detection (Det) improves performance across all object types, especially when the target is small or far away, with mean SPL increasing from 0.28 to 0.52 without and with Det.
• Suboptimal behaviors such as revisiting explored regions or getting stuck in blind spots emerge, indicating limitations in long-term memory and perception beyond field of view.