[Paper Review] Learning Object-specific Distance from a Monocular Image

Jing Zhu, Yi Fang+4 more|arXiv (Cornell University)|Sep 9, 2019

Video Surveillance and Tracking Methods33 references4 citations

TL;DR

This paper proposes the first end-to-end deep learning model for object-specific distance estimation from monocular RGB images in autonomous driving, using a base model with ROI pooling and a refined enhanced model with a keypoint regressor and projection loss for improved accuracy, especially near the camera. The method outperforms traditional IPM and SVR baselines on extended KITTI and nuScenes (mini) datasets, achieving lower RMSE and higher accuracy, particularly on curved roads and long-range objects.

ABSTRACT

Environment perception, including object detection and distance estimation, is one of the most crucial tasks for autonomous driving. Many attentions have been paid on the object detection task, but distance estimation only arouse few interests in the computer vision community. Observing that the traditional inverse perspective mapping algorithm performs poorly for objects far away from the camera or on the curved road, in this paper, we address the challenging distance estimation problem by developing the first end-to-end learning-based model to directly predict distances for given objects in the images. Besides the introduction of a learning-based base model, we further design an enhanced model with a keypoint regressor, where a projection loss is defined to enforce a better distance estimation, especially for objects close to the camera. To facilitate the research on this task, we construct the extented KITTI and nuScenes (mini) object detection datasets with a distance for each object. Our experiments demonstrate that our proposed methods outperform alternative approaches (e.g., the traditional IPM, SVR) on object-specific distance estimation, particularly for the challenging cases that objects are on a curved road. Moreover, the performance margin implies the effectiveness of our enhanced method.

Motivation & Objective

• To address the underexplored challenge of object-specific distance estimation from monocular images in autonomous driving.
• To overcome the limitations of traditional inverse perspective mapping (IPM), which fails for distant or side-facing objects and on curved roads.
• To develop a deep learning-based approach that directly predicts object distances without relying on camera parameters during inference.
• To improve estimation accuracy, especially for objects close to the camera, through a novel keypoint-based projection loss.
• To construct a new benchmark dataset by extending KITTI and nuScenes (mini) with ground-truth distances derived from LiDAR and camera calibration.

Proposed method

• Proposes a base model that uses a CNN (e.g., VGG16 or ResNet50) to extract features from RGB images, applies ROI pooling to generate fixed-size features per object, and regresses distance using a fully connected layer.
• Introduces an enhanced model with a keypoint regressor that predicts 2D image coordinates (X, Y) of a 3D object keypoint, combined with predicted Z-distance to form a 3D point.
• Defines a projection loss that enforces the 3D predicted keypoint to project back to the ground-truth 2D keypoint location using the camera projection matrix, improving spatial consistency.
• Trains the model end-to-end using a combined loss of distance regression loss and projection loss, with the classifier branch to learn category-discriminative features.
• Uses camera parameters only during training to compute the projection loss; inference is free of camera parameters, enabling generalization across visual systems.
• Constructs an extended dataset by computing object distances from LiDAR point clouds and camera parameters, adding ground-truth distances to KITTI and nuScenes (mini) object detection annotations.

Experimental results

Research questions

• RQ1 Can a deep learning model directly predict object-specific distances from a single RGB image without relying on camera parameters during inference?
• RQ2 How does the inclusion of a keypoint regressor with projection loss improve distance estimation accuracy, especially for near-camera objects?
• RQ3 Can the proposed model generalize to challenging scenarios such as curved roads and long-range object detection where traditional IPM fails?
• RQ4 How does the model compare quantitatively to classical baselines like IPM and SVR in terms of distance estimation accuracy and inference speed?
• RQ5 To what extent does adding a category classifier improve the model’s ability to estimate object distances?

Key findings

• The enhanced model achieves a δ1 score of 0.629 on the KITTI dataset, significantly outperforming the base model (0.587) and SVR (0.308), demonstrating the effectiveness of the keypoint projection loss.
• On the nuScenes (mini) dataset, the enhanced model achieves a δ1 of 0.535 and RMSE of 10.511, outperforming IPM (δ1: 0.441, RMSE: 249.849) and SVR (δ1: 0.308, RMSE: 18.480).
• The enhanced model reduces RMSE log from 0.512 (base model) to 0.314 on KITTI, indicating improved performance on long-range distance estimation.
• Inference time is 16.2ms per image with VGG16, which is twice as fast as IPM (33.9ms) and faster than SVR (12.1ms), showing good real-time feasibility.
• The inclusion of a classifier improves the AR (average relative error) from 0.658 (base w/o classifier) to 0.311 (base w classifier), proving its value in learning category-aware features.
• Qualitative results show that the model successfully estimates distances for objects beyond 40 meters and on curved roads, where IPM fails, while maintaining high accuracy for near objects.