[Paper Review] ClusterVO: Clustering Moving Instances and Estimating Visual Odometry for Self and Surroundings

Jiahui Huang, Sheng Yang+2 more|arXiv (Cornell University)|Mar 29, 2020

Video Surveillance and Tracking Methods47 references4 citations

TL;DR

ClusterVO proposes a real-time stereo visual odometry system that jointly estimates ego-motion and clusters moving rigid objects in dynamic scenes without relying on geometric or object-specific priors. It uses a multi-level probabilistic association and a heterogeneous CRF integrating semantic, spatial, and motion cues to enable online clustering and sliding-window optimization, achieving state-of-the-art performance on KITTI and Oxford Multimotion datasets with 8 FPS and competitive accuracy.

ABSTRACT

We present ClusterVO, a stereo Visual Odometry which simultaneously clusters and estimates the motion of both ego and surrounding rigid clusters/objects. Unlike previous solutions relying on batch input or imposing priors on scene structure or dynamic object models, ClusterVO is online, general and thus can be used in various scenarios including indoor scene understanding and autonomous driving. At the core of our system lies a multi-level probabilistic association mechanism and a heterogeneous Conditional Random Field (CRF) clustering approach combining semantic, spatial and motion information to jointly infer cluster segmentations online for every frame. The poses of camera and dynamic objects are instantly solved through a sliding-window optimization. Our system is evaluated on Oxford Multimotion and KITTI dataset both quantitatively and qualitatively, reaching comparable results to state-of-the-art solutions on both odometry and dynamic trajectory recovery.

Motivation & Objective

• To develop a general-purpose, real-time visual odometry system that simultaneously estimates ego-motion and tracks multiple moving rigid objects in dynamic scenes.
• To overcome limitations of prior methods that rely on batch processing, scene priors, or specialized assumptions about object shape or structure.
• To enable accurate and consistent clustering of dynamic objects and camera motion in both indoor and outdoor environments, including autonomous driving and AR applications.
• To design an online system that maintains high efficiency and robustness without requiring dense RGB-D input or pre-defined object templates.

Proposed method

• A multi-level probabilistic association mechanism tracks sparse 2D detections and 3D landmarks across frames using motion consistency and spatial proximity.
• A heterogeneous Conditional Random Field (CRF) integrates semantic bounding boxes, spatial affinity, and motion consistency to jointly infer cluster segmentations in real time.
• The system employs a sliding-window optimization framework to simultaneously solve for camera pose and dynamic object trajectories.
• The CRF formulation combines unary and pairwise potentials, with terms for semantic confidence, spatial clustering, and motion coherence (Eq. 5).
• The method avoids reliance on object size priors or scene structure assumptions, enabling generalization across diverse environments.
• The system is trained and evaluated on stereo sequences from KITTI and Oxford Multimotion datasets, using only sparse features and 2D detections.

Experimental results

Research questions

• RQ1 Can a general-purpose visual odometry system achieve accurate ego-motion and dynamic object tracking without scene-specific priors?
• RQ2 How can online clustering of moving rigid objects be achieved robustly using only sparse 2D detections and 3D landmarks?
• RQ3 What is the impact of combining semantic, spatial, and motion cues in a heterogeneous CRF for dynamic scene understanding?
• RQ4 Can a real-time system achieve performance comparable to batch-optimized or specialized methods in both odometry and dynamic trajectory recovery?

Key findings

• ClusterVO achieves 8 FPS on the KITTI dataset, significantly outperforming methods like Chen et al. (1.2s per frame) and matching the efficiency of real-time systems like DynSLAM.
• On the KITTI dataset, ClusterVO achieves a translation error of 0.52 m (ATE) and rotation error of 0.19 rad (RPE) for ego-motion, comparable to state-of-the-art methods.
• For dynamic object tracking, ClusterVO achieves 49.65% AP (Average Precision) on the KITTI 'Moderate' category, outperforming DynSLAM (47.16%) and showing robustness to detection misses.
• Ablation studies show that adding 3D geometric and semantic cues to the CRF reduces cluster motion ATE by 45.8% compared to 2D-only CRF, demonstrating the effectiveness of multi-modal fusion.
• The system achieves competitive results on the Oxford Multimotion dataset, with ego-motion ATE of 0.61 m and cluster motion ATE of 0.13 m, approaching the performance of batch-optimized ClusterSLAM despite being online.
• Visualizations confirm that the heterogeneous CRF effectively filters misclassified landmarks, improving clustering consistency and trajectory estimation accuracy.