[Paper Review] Self-supervised learning for autonomous vehicles perception: A conciliation between analytical and learning methods

Florent Chiaroni, Mohamed-Cherif Rahal+2 more|arXiv (Cornell University)|Oct 3, 2019

Robotics and Sensor-Based LocalizationEngineering73 references18 citations

TL;DR

This paper proposes a self-supervised learning (SSL) framework that unifies analytical and deep learning methods for autonomous vehicle perception, enabling end-to-end training without hand-labeled data. By leveraging spatio-temporal or multi-sensor data to generate pseudo-labels (e.g., depth, motion, segmentation), the method achieves competitive performance on tasks like depth estimation and obstacle detection, with online adaptation and self-evaluation capabilities.

ABSTRACT

Nowadays, supervised deep learning techniques yield the best state-of-the-art prediction performances for a wide variety of computer vision tasks. However, such supervised techniques generally require a large amount of manually labeled training data. In the context of autonomous vehicles perception, this requirement is critical, as the distribution of sensor data can continuously change and include several unexpected variations. It turns out that a category of learning techniques, referred to as self-supervised learning (SSL), consists of replacing the manual labeling effort by an automatic labeling process. Thanks to their ability to learn on the application time and in varying environments, state-of-the-art SSL techniques provide a valid alternative to supervised learning for a variety of different tasks, including long-range traversable area segmentation, moving obstacle instance segmentation, long-term moving obstacle tracking, or depth map prediction. In this tutorial-style article, we present an overview and a general formalization of the concept of self-supervised learning (SSL) for autonomous vehicles perception. This formalization provides helpful guidelines for developing novel frameworks based on generic SSL principles. Moreover, it enables to point out significant challenges in the design of future SSL systems.

Motivation & Objective

• Address the high cost and scarcity of fully annotated datasets for autonomous vehicle perception.
• Overcome limitations of purely analytical methods in handling high-dimensional, complex semantic patterns in urban scenes.
• Mitigate the need for large-scale supervised data while improving model robustness to unseen scenarios.
• Enable online self-evaluation, self-improvement, and adaptation in real-time driving environments.
• Integrate multimodal sensor data (e.g., stereo, temporal sequences) to generate reliable pseudo-labels for self-supervised training.

Proposed method

• Use a single monocular camera frame as input and predict a downstream task (e.g., depth, segmentation, motion) using auxiliary information from temporal sequences or stereo pairs as pseudo-labels.
• Apply a learning model (e.g., CNN or transformer) to predict the target task from a single frame, trained via a reconstruction or consistency loss against the pseudo-labeled data.
• Incorporate analytical methods such as optical flow, epipolar geometry, or stereo triangulation to generate ground-truth-like supervision without human annotation.
• Use re-projection loss and motion consistency to filter out dynamic objects and occluded regions during training.
• Enable online learning by continuously updating the model with new data, allowing adaptation to novel environments.
• Integrate self-evaluation mechanisms using analytical baselines to detect model uncertainty or prediction errors.

Experimental results

Research questions

• RQ1 Can self-supervised learning effectively replace fully supervised training for autonomous vehicle perception tasks without human-annotated data?
• RQ2 How can analytical methods be systematically integrated with deep learning to improve robustness and interpretability in perception systems?
• RQ3 To what extent can SSL frameworks enable online adaptation and self-evaluation in real-world driving scenarios?
• RQ4 What are the limitations of SSL in handling dynamic obstacles and depth scale ambiguity, and how can they be mitigated?
• RQ5 How can catastrophic forgetting be prevented during continuous online learning in autonomous driving perception?

Key findings

• Self-supervised learning methods achieve state-of-the-art performance on depth estimation tasks, with results competitive to fully supervised models—e.g., [56]* > [47] > [16] > [54] on KITTI.
• SSL frameworks based on temporal sequences and stereo data can generate accurate pseudo-labels for depth and motion estimation without ground-truth annotations.
• The integration of analytical methods (e.g., epipolar geometry, optical flow) enables robust handling of occlusions and moving objects during training.
• Online self-evaluation is feasible using analytical baselines, though noise in these baselines may affect reliability and requires further investigation.
• Catastrophic forgetting remains a challenge in continuous learning, but incremental learning techniques offer a viable path forward.
• The scale ambiguity in monocular depth estimation persists, but can be addressed by estimating metric pose transformations or using template matching for scale recovery.