[Paper Review] Joint 3D Reconstruction of a Static Scene and Moving Objects

Sergio Caccamo, Esra Ataer-Cansızoğlu+1 more|arXiv (Cornell University)|Feb 13, 2018

Robotics and Sensor-Based Localization24 references3 citations

TL;DR

This paper proposes a real-time SLAM system that jointly reconstructs static scenes and rigidly moving objects using RGB-D data by maintaining separate, independent maps for static and dynamic regions. It employs a multi-group registration scheme—fusing full-frame and segment-based RANSAC registrations—to robustly classify features as static or dynamic, enabling on-the-fly 3D object model generation without prior assumptions about motion patterns.

ABSTRACT

We present a technique for simultaneous 3D reconstruction of static regions and rigidly moving objects in a scene. An RGB-D frame is represented as a collection of features, which are points and planes. We classify the features into static and dynamic regions and grow separate maps, static and object maps, for each of them. To robustly classify the features in each frame, we fuse multiple RANSAC-based registration results obtained by registering different groups of the features to different maps, including (1) all the features to the static map, (2) all the features to each object map, and (3) subsets of the features, each forming a segment, to each object map. This multi-group registration approach is designed to overcome the following challenges: scenes can be dominated by static regions, making object tracking more difficult; and moving object might have larger pose variation between frames compared to the static regions. We show qualitative results from indoor scenes with objects in various shapes. The technique enables on-the-fly object model generation to be used for robotic manipulation.

Motivation & Objective

• Address the challenge of simultaneous 3D reconstruction of static environments and rigidly moving objects in dynamic scenes.
• Overcome the chicken-and-egg problem in SLAM where accurate camera localization requires known object motion, and vice versa.
• Enable on-the-fly 3D object model generation during scene mapping without requiring separate scanning or smooth motion.
• Maintain robust feature classification between static and dynamic regions despite dominant static structures or high object pose variation.
• Support both continuous and discrete object motion patterns, including objects appearing in non-consecutive frames.

Proposed method

• Represent each RGB-D frame as a collection of 3D points, 2D points, and 3D planes, grouped into segments for geometric analysis.
• Initialize a static map using the first frame and detect moving objects via high outlier ratios after frame registration to the static map.
• Use multi-group registration: register all features and various feature subsets (segments) to both static and object maps to improve robustness.
• Fuse multiple RANSAC-based registration results to classify features as belonging to static or dynamic regions based on inlier consistency.
• Maintain separate, disjoint maps for static and moving objects, enabling independent optimization and stable long-term tracking.
• Perform asynchronous bundle adjustment on both maps to refine 3D structure and camera poses using sparse feature representations.

Experimental results

Research questions

• RQ1 How can a SLAM system simultaneously reconstruct static scenes and rigidly moving objects without prior knowledge of object motion?
• RQ2 What geometric verification strategy enables robust feature classification between static and dynamic regions in the presence of dominant static structures?
• RQ3 Can on-the-fly 3D object model construction be achieved during autonomous exploration without requiring dedicated scanning phases?
• RQ4 How does multi-group registration improve robustness compared to single-registration approaches in dynamic scenes?
• RQ5 What are the limitations of the system when multiple moving objects are present in the same frame, and how can they be mitigated?

Key findings

• The multi-group registration approach significantly improves feature classification accuracy by combining full-frame and segment-based RANSAC results, especially when static regions dominate the scene.
• The system successfully reconstructs complete 3D object models from partial, occluded observations across multiple keyframes, as demonstrated in Figure 6(D).
• The static map grows steadily over time, while the object map only expands when the object is visible, confirming effective dynamic region detection (Figure 7).
• The method enables real-time, on-the-fly 3D object modeling without assuming smooth or continuous motion, supporting both continuous and discrete object appearances.
• The system can handle scenes with abrupt object motion and texture-less objects, though performance degrades in low-texture regions due to sparse feature reliance.
• A key limitation is the inability to distinguish multiple moving objects in a single frame, leading to a single shared object map, which may hinder accurate reconstruction.