[Paper Review] ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection

Zhenbo Xu, Wei Zhang+7 more|arXiv (Cornell University)|Mar 1, 2020

Advanced Neural Network Applications19 references4 citations

TL;DR

ZoomNet is a novel stereo 3D object detection framework that uses part-aware adaptive zooming to enhance disparity estimation and 3D point cloud reconstruction for distant and occluded objects. By resizing object instances to a uniform resolution and adjusting camera parameters, it improves point cloud density and quality, achieving state-of-the-art performance with a 9.4% APbv (IoU=0.7) gain over pseudo-LiDAR on KITTI.

ABSTRACT

3D object detection is an essential task in autonomous driving and robotics. Though great progress has been made, challenges remain in estimating 3D pose for distant and occluded objects. In this paper, we present a novel framework named ZoomNet for stereo imagery-based 3D detection. The pipeline of ZoomNet begins with an ordinary 2D object detection model which is used to obtain pairs of left-right bounding boxes. To further exploit the abundant texture cues in RGB images for more accurate disparity estimation, we introduce a conceptually straight-forward module -- adaptive zooming, which simultaneously resizes 2D instance bounding boxes to a unified resolution and adjusts the camera intrinsic parameters accordingly. In this way, we are able to estimate higher-quality disparity maps from the resized box images then construct dense point clouds for both nearby and distant objects. Moreover, we introduce to learn part locations as complementary features to improve the resistance against occlusion and put forward the 3D fitting score to better estimate the 3D detection quality. Extensive experiments on the popular KITTI 3D detection dataset indicate ZoomNet surpasses all previous state-of-the-art methods by large margins (improved by 9.4% on APbv (IoU=0.7) over pseudo-LiDAR). Ablation study also demonstrates that our adaptive zooming strategy brings an improvement of over 10% on AP3d (IoU=0.7). In addition, since the official KITTI benchmark lacks fine-grained annotations like pixel-wise part locations, we also present our KFG dataset by augmenting KITTI with detailed instance-wise annotations including pixel-wise part location, pixel-wise disparity, etc.. Both the KFG dataset and our codes will be publicly available at https://github.com/detectRecog/ZoomNet.

Motivation & Objective

• Address the challenge of inaccurate 3D detection for distant and occluded objects in stereo imagery-based systems.
• Overcome the limitations of holistic disparity estimation in pseudo-LiDAR, which degrades for faraway or occluded objects due to low resolution and sparse features.
• Improve robustness to occlusion by introducing pixel-wise part location supervision as complementary 3D cues.
• Enhance 3D detection confidence estimation by proposing a 3D fitting score, which correlates better with 3D IoU than 2D classification probability.
• Develop a new dataset, KFG, with fine-grained annotations (pixel-wise part locations, disparity) to support future research on stereo 3D detection.

Proposed method

• Apply a standard 2D object detector to generate left-right instance bounding boxes for each object.
• Implement adaptive zooming: resize each 2D instance to a fixed resolution and scale the camera intrinsic parameters accordingly to maintain geometric consistency.
• Estimate high-resolution disparity maps on the zoomed instance images, enabling more accurate depth estimation for distant objects.
• Project the zoomed disparity maps into dense 3D point clouds using the adjusted camera parameters, resulting in uniformly dense point clouds regardless of object distance.
• Introduce a part location head to predict pixel-wise part locations (e.g., front, rear, side) as 3D-aware supervision to improve generalization under occlusion.
• Propose a 3D fitting score as a confidence measure for 3D detection, computed based on 3D IoU between prediction and ground truth, to better reflect detection quality than 2D classification scores.

Experimental results

Research questions

• RQ1 Can adaptive zooming of object instances improve disparity estimation and 3D point cloud quality for distant objects in stereo 3D detection?
• RQ2 How does incorporating pixel-wise part location supervision enhance detection robustness under occlusion?
• RQ3 Does a 3D fitting score provide a better confidence estimation for 3D detection than 2D classification probability?
• RQ4 To what extent does the combination of adaptive zooming, part location, and 3D fitting score improve performance on challenging KITTI benchmarks?
• RQ5 Can a new fine-grained dataset (KFG) with pixel-wise annotations enable better evaluation and training for stereo 3D detection?

Key findings

• ZoomNet achieves a 9.4% absolute improvement in APbv (IoU=0.7) over pseudo-LiDAR on the KITTI 3D detection benchmark, setting a new SOTA.
• On the KITTI Mode set, ZoomNet achieves 72.94% APbv (IoU=0.7), surpassing all prior image-based methods and marking the first time a stereo-based method exceeds 70% in the Hard set.
• The ablation study shows that adaptive zooming alone improves AP3d (IoU=0.7) by over 10% compared to baseline, demonstrating its critical role in enhancing point cloud density and quality.
• Removing part location supervision causes a 40.88% drop in AP3d (IoU=0.7) on the Hard set, confirming its importance for detecting occluded objects.
• The 3D fitting score significantly improves detection confidence estimation, with a 4% drop in AP3d (IoU=0.7) when removed, especially at high IoU thresholds.
• The KFG dataset, released alongside the code, provides pixel-wise annotations for part locations and disparity, enabling future research on fine-grained 3D detection.