[Paper Review] Auto4D: Learning to Label 4D Objects from Sequential Point Clouds

Bin Yang, Min Bai+3 more|arXiv (Cornell University)|Jan 17, 2021

Advanced Neural Network ApplicationsComputer Science42 references25 citations

TL;DR

Auto4D automatically generates high-quality 4D (3D+time) vehicle labels by decomposing the problem into fixed object size estimation and motion path refinement over full trajectories, reducing human annotation effort by up to 25%.

ABSTRACT

In the past few years we have seen great advances in object perception (particularly in 4D space-time dimensions) thanks to deep learning methods. However, they typically rely on large amounts of high-quality labels to achieve good performance, which often require time-consuming and expensive work by human annotators. To address this we propose an automatic annotation pipeline that generates accurate object trajectories in 3D space (i.e., 4D labels) from LiDAR point clouds. The key idea is to decompose the 4D object label into two parts: the object size in 3D that's fixed through time for rigid objects, and the motion path describing the evolution of the object's pose through time. Instead of generating a series of labels in one shot, we adopt an iterative refinement process where online generated object detections are tracked through time as the initialization. Given the cheap but noisy input, our model produces higher quality 4D labels by re-estimating the object size and smoothing the motion path, where the improvement is achieved by exploiting aggregated observations and motion cues over the entire trajectory. We validate the proposed method on a large-scale driving dataset and show a 25% reduction of human annotation efforts. We also showcase the benefits of our approach in the annotator-in-the-loop setting.

Motivation & Objective

• Motivate automatic 4D labeling to reduce human annotation cost for autonomous driving datasets.
• Propose a two-branch model that separately estimates a constant 3D size and refines the motion trajectory over time.
• Leverage full trajectory observations to improve 3D bounding box accuracy (IoU) and trajectory smoothness.
• Evaluate on a high-quality Car4D dataset to demonstrate substantial improvements over baselines.

Proposed method

• Use initial online detector + discrete tracker to obtain noisy 4D object trajectories.
• Object size branch aggregates multi-frame observations to predict a single constant size for each object and refines boxes with a corner-align strategy.
• Motion path branch uses a spatial-temporal encoder–decoder to refine the pose trajectory using 4D point clouds and motion cues.
• Train branches sequentially with IoU-based loss; during inference apply size refinement across the trajectory and then sliding-window path refinement.

Experimental results

Research questions

• RQ1 Can leveraging observations over the entire object trajectory yield more accurate constant-size estimates for 3D bounding boxes?
• RQ2 Does incorporating a motion-path refinement over the full trajectory improve 4D labeling precision beyond static size estimation?
• RQ3 How much can automatic 4D labeling reduce the need for human correction in high-quality driving datasets?
• RQ4 Is the corner-align strategy for size refinement superior to center-aligned approaches under LiDAR sparsity and occlusions?

Key findings

• Auto4D increases the number of boxes with IoU ≥ 0.9 by 8.4% from the size branch and by 6.3% from the path branch on Car4D test set.
• Overall, Auto4D yields about a 25% reduction in human effort to correct poorly localized boxes (IoU ≥ 0.9) compared to the online detector + discrete tracker baseline.
• The size branch with corner-align refinement significantly outperforms center-align and random baselines in producing precise size estimates.
• The annotator-in-the-loop experiment demonstrates that minimal human corrections can further improve labeling accuracy without retraining the model.
• Static objects benefit more from the size branch, while moving objects gain more from the motion path refinement.