[Paper Review] Geometry-Aware Instance Segmentation with Disparity Maps

Cho-Ying Wu, Xiaoyan Hu+3 more|arXiv (Cornell University)|Jun 14, 2020

Advanced Vision and Imaging42 references4 citations

TL;DR

This paper proposes GAIS-Net, a geometry-aware instance segmentation framework that fuses RGB images and disparity maps from stereo cameras to improve 3D-aware object detection and segmentation in autonomous driving. By leveraging pseudo-lidar projections and multi-modal ROI features (2D, 2.5D, 3D), the method enhances mask prediction and suppresses false positives through geometric priors, achieving state-of-the-art performance on the newly collected High-Quality Driving Stereo (HQDS) dataset with a 4× larger baseline and focal length than Cityscapes.

ABSTRACT

Most previous works of outdoor instance segmentation for images only use color information. We explore a novel direction of sensor fusion to exploit stereo cameras. Geometric information from disparities helps separate overlapping objects of the same or different classes. Moreover, geometric information penalizes region proposals with unlikely 3D shapes thus suppressing false positive detections. Mask regression is based on 2D, 2.5D, and 3D ROI using the pseudo-lidar and image-based representations. These mask predictions are fused by a mask scoring process. However, public datasets only adopt stereo systems with shorter baseline and focal legnth, which limit measuring ranges of stereo cameras. We collect and utilize High-Quality Driving Stereo (HQDS) dataset, using much longer baseline and focal length with higher resolution. Our performance attains state of the art. Please refer to our project page. The full paper is available here.

Motivation & Objective

• To improve instance segmentation robustness in outdoor autonomous driving by integrating geometric cues from stereo disparity maps.
• To address limitations of monocular or RGB-only methods that are sensitive to lighting, shadows, and texture variations.
• To develop a multi-modal, end-to-end trainable network that fuses 2D, 2.5D, and 3D representations for improved mask prediction.
• To collect and release a high-quality stereo dataset (HQDS) with longer baseline and focal length to enable better far-field stereo matching and geometric reasoning.
• To demonstrate that geometric priors from disparity maps reduce false positives and improve mask quality, especially for overlapping or distant objects.

Proposed method

• The method uses PSMNet to generate dense disparity maps from stereo image pairs, which are projected into 3D space via pseudo-lidar back-projection.
• Region proposals are extracted from the left image using a ResNet50-FPN backbone and RPN, forming 2D ROIs.
• 2.5D ROIs are cropped from the disparity map, while 3D ROIs are created by back-projecting disparity values into 3D point clouds.
• A PointNet-based network processes 3D ROIs to extract 3D features and predict per-point mask probabilities, which are re-projected to 2D for loss computation.
• 2D, 2.5D, and 3D mask predictions are fused via a mask scoring mechanism that improves final mask confidence and IoU.
• The network is trained with combined losses: $π_{box}$, $π_{cls}$, $π_{2Dmask}$, $π_{3Dmask}$, and a MaskIoU head to refine mask quality.

Experimental results

Research questions

• RQ1 Can disparity maps from stereo cameras improve instance segmentation performance by providing geometric priors for shape and depth?
• RQ2 How does fusing 2D, 2.5D, and 3D representations enhance mask prediction accuracy and reduce false positives?
• RQ3 To what extent does a longer baseline and focal length in stereo systems improve far-field object detection and geometric consistency?
• RQ4 Can a multi-modal network that combines image, disparity, and 3D point cloud features outperform purely RGB-based instance segmentation models?
• RQ5 How does the proposed mask scoring fusion mechanism improve final segmentation quality compared to individual modality predictions?

Key findings

• GAIS-Net achieves state-of-the-art performance on the HQDS dataset, with a mask AP of 40.7% compared to 33.9% for Mask-RCNN using the same backbone.
• The method improves bounding box AP by 9.7 percentage points (from 36.3% to 46.0%) and mask AP by 6.8 percentage points (from 33.9% to 40.7%) over Mask-RCNN on the HQDS test set.
• On Cityscapes, GAIS-Net achieves 37.1% mask AP when pre-trained on COCO, outperforming Mask-RCNN’s 36.4% under the same setup.
• The performance gap between HQDS and Cityscapes is primarily due to HQDS’s longer baseline (0.5 m) and focal length (3.3K px), which enable better far-field stereo matching and geometric estimation.
• The mask scoring fusion mechanism improves mask quality by integrating predictions from 2D, 2.5D, and 3D representations, leading to more consistent and accurate masks.
• The HQDS dataset, with 6K training and 1.2K testing stereo pairs at 1024×3072 resolution and a 4× larger $f \times b$ product than Cityscapes, enables more reliable evaluation of stereo-based instance segmentation.