[Paper Review] FISHING Net: Future Inference of Semantic Heatmaps In Grids

Noureldin Hendy, Cooper Stokes Sloan+6 more|arXiv (Cornell University)|Jun 17, 2020

Distributed and Parallel Computing Systems17 references51 citations

TL;DR

The paper presents an end-to-end multi-modal framework that predicts current and near-future top-down semantic grids from lidar, radar, and camera data, enabling unified cross-modal perception and short-term prediction.

ABSTRACT

For autonomous robots to navigate a complex environment, it is crucial to understand the surrounding scene both geometrically and semantically. Modern autonomous robots employ multiple sets of sensors, including lidars, radars, and cameras. Managing the different reference frames and characteristics of the sensors, and merging their observations into a single representation complicates perception. Choosing a single unified representation for all sensors simplifies the task of perception and fusion. In this work, we present an end-to-end pipeline that performs semantic segmentation and short term prediction using a top-down representation. Our approach consists of an ensemble of neural networks which take in sensor data from different sensor modalities and transform them into a single common top-down semantic grid representation. We find this representation favorable as it is agnostic to sensor-specific reference frames and captures both the semantic and geometric information for the surrounding scene. Because the modalities share a single output representation, they can be easily aggregated to produce a fused output. In this work we predict short-term semantic grids but the framework can be extended to other tasks. This approach offers a simple, extensible, end-to-end approach for multi-modal perception and prediction.

Motivation & Objective

• Unify multi-sensor observations into a single, sensor-agnostic top-down semantic grid representation.
• Enable simultaneous perception and short-term prediction within one end-to-end pipeline.
• Assess the benefits of different fusion strategies for multi-modal semantic grid outputs.
• Demonstrate extensibility of the framework to diverse sensor modalities and reference frames.

Proposed method

• An ensemble of modality-specific encoder-decoder networks outputs a sequence of top-down semantic grids for each sensor.
• Vision uses a pixel-space-to-top-down learned orthographic feature transform between encoder and decoder.
• Lidar and radar employ a U-Net-like encoder-decoder to predict grids.
• Vision features from multiple cameras are fused after a shared encoder with a learned orthographic transform.
• Multi-modal fusion aggregates softmax outputs across modalities using average pooling or a priority-based pool.

Experimental results

Research questions

• RQ1 Can multiple sensor modalities (lidar, radar, camera) be mapped to a common top-down semantic grid representation?
• RQ2 Does end-to-end fusion of current and near-future semantic grids improve perception and short-term prediction for autonomous navigation?
• RQ3 How do different fusion strategies (average vs. priority pool) affect perception accuracy across modalities?
• RQ4 Is the approach extensible to additional modalities and reference frames without changing the output grid representation?
• RQ5 What is the comparative performance of the framework on open datasets (NuScenes, Lyft) and a purpose-built dataset?

Key findings

• The framework can predict short-term future semantic grids (up to 2s) for three classes: VRU, vehicles, and background.
• Lidar generally yields higher precision and recall than vision on NuScenes and Lyft datasets for current and near-future grids.
• Vision-based top-down predictions are feasible but more challenging due to depth and ego-motion estimation from monocular cues.
• Average pooling generally yields higher precision, while priority pooling yields higher recall across datasets.
• Radar contributes complementary depth/kinematics information and improves performance in the purpose-built dataset where it is included.
• The approach scales with larger, richer datasets and shows better performance with more modalities (Lidar > Radar > Vision in reported results).