[Paper Review] The StreetLearn Environment and Dataset

Piotr Mirowski, Andras Banki-Horvath+9 more|arXiv (Cornell University)|Mar 4, 2019

Multimodal Machine Learning Applications26 references49 citations

TL;DR

The paper introduces StreetLearn, an interactive, first-person navigation environment using Google Street View content, and provides baselines for a courier navigation task across multiple city regions. It also releases code and a scalable evaluation framework for end-to-end navigation with visual input.

ABSTRACT

Navigation is a rich and well-grounded problem domain that drives progress in many different areas of research: perception, planning, memory, exploration, and optimisation in particular. Historically these challenges have been separately considered and solutions built that rely on stationary datasets - for example, recorded trajectories through an environment. These datasets cannot be used for decision-making and reinforcement learning, however, and in general the perspective of navigation as an interactive learning task, where the actions and behaviours of a learning agent are learned simultaneously with the perception and planning, is relatively unsupported. Thus, existing navigation benchmarks generally rely on static datasets (Geiger et al., 2013; Kendall et al., 2015) or simulators (Beattie et al., 2016; Shah et al., 2018). To support and validate research in end-to-end navigation, we present StreetLearn: an interactive, first-person, partially-observed visual environment that uses Google Street View for its photographic content and broad coverage, and give performance baselines for a challenging goal-driven navigation task. The environment code, baseline agent code, and the dataset are available at http://streetlearn.cc

Motivation & Objective

• Motivate end-to-end visual navigation in real-world-like environments beyond static datasets.
• Present StreetLearn as an interactive, first-person navigation environment grounded in Google Street View imagery.
• Define delivery-style courier tasks and instruction-following tasks to probe navigation policies.
• Provide a scalable benchmark with region-based evaluation, curricula, and transferable agent architectures.

Proposed method

• Construct StreetLearn from Google Street View panoramas forming a real-world street graph for two cities (New York City and Pittsburgh).
• Define observation space as 84x84 RGB crops and a five-action discrete/six discrete action set for rotation, movement, and zoom.
• Formalize tasks including a courier goal-following task and instruction-based navigation with absolute coordinate goals or language-guided instructions.
• Propose two neural architectures (CityNav and MultiCityNav) with shared encoders and city-specific LSTMs, trained with IMPALA for scalable RL.
• Provide an oracle baseline via shortest-path BFS on the street graph to bound performance.
• Release codebase with C++ engine, protocol buffers, Python gym-like interface, and TensorFlow agents.

Experimental results

Research questions

• RQ1 Can end-to-end navigation policies be learned directly from visual input in real-world-like Street View graphs?
• RQ2 How well do region-specific and multi-city architectures generalize and transfer across different urban regions?
• RQ3 What is the impact of curriculum learning and goal representations on long-range navigation performance?
• RQ4 How does imitation/ground-truth guidance (oracle) bound the performance of learned policies?
• RQ5 Does goal specification (absolute lat/long vs. landmarks) affect navigation efficacy?

Key findings

• Agents achieve 85%-97% of oracle returns in New York regions when trained per region.
• Performance drops in Pittsburgh regions (notably South Shore) due to elevation and road topology affecting curriculum design.
• Joint training across multiple regions and cities yields only a small drop in performance compared to region-specific training.
• Transfer experiments show that freezing encoder/policy components and updating only the goal LSTM allows transfer to a new region with moderate loss.
• Oracle (shortest-path) provides an upper bound on achievable performance for each region.
• Lat/long target representations outperform landmark-based targets in at least one region (Union Square).