[Paper Review] ROSEFusion: Random Optimization for Online Dense Reconstruction under Fast Camera Motion

Jiazhao Zhang, Chenyang Zhu+2 more|arXiv (Cornell University)|May 12, 2021

Advanced Vision and Imaging48 references217 citations

TL;DR

ROSEFusion pursues real-time dense RGB-D reconstruction under fast camera motion by using a particle filter optimization with a pre-sampled particle swarm template and a depth-based fitness function, without inertial data.

ABSTRACT

Online reconstruction based on RGB-D sequences has thus far been restrained to relatively slow camera motions (<1m/s). Under very fast camera motion (e.g., 3m/s), the reconstruction can easily crumble even for the state-of-the-art methods. Fast motion brings two challenges to depth fusion: 1) the high nonlinearity of camera pose optimization due to large inter-frame rotations and 2) the lack of reliably trackable features due to motion blur. We propose to tackle the difficulties of fast-motion camera tracking in the absence of inertial measurements using random optimization, in particular, the Particle Filter Optimization (PFO). To surmount the computation-intensive particle sampling and update in standard PFO, we propose to accelerate the randomized search via updating a particle swarm template (PST). PST is a set of particles pre-sampled uniformly within the unit sphere in the 6D space of camera pose. Through moving and rescaling the pre-sampled PST guided by swarm intelligence, our method is able to drive tens of thousands of particles to locate and cover a good local optimum extremely fast and robustly. The particles, representing candidate poses, are evaluated with a fitness function defined based on depth-model conformance. Therefore, our method, being depth-only and correspondence-free, mitigates the motion blur impediment as ToF-based depths are often resilient to motion blur. Thanks to the efficient template-based particle set evolution and the effective fitness function, our method attains good quality pose tracking under fast camera motion (up to 4m/s) in a realtime framerate without including loop closure or global pose optimization. Through extensive evaluations on public datasets of RGB-D sequences, especially on a newly proposed benchmark of fast camera motion, we demonstrate the significant advantage of our method over the state of the arts.

Motivation & Objective

• Address online dense RGB-D reconstruction under fast camera motion without inertial measurements.
• Propose a robust depth-only camera tracking method that handles large inter-frame rotations and motion blur.
• Introduce a fast, template-based random optimization to replace standard particle sampling.
• Provide a depth-to-TSDF fitness function to evaluate candidate poses.
• Offer a new fast-motion RGB-D benchmark and demonstrate competitive performance against state-of-the-art methods.

Proposed method

• Formulates per-frame pose optimization as depth-to-TSDF conformance using a frame-to-model likelihood.
• Introduces Particle Filter Optimization (PFO) for poses with a pre-sampled Particle Swarm Template (PST).
• Evolves PST by moving and resizing a 6D ellipsoidal template to cover good local optima while evaluating candidates with a depth-based fitness function.
• Defines the fitness as the exponential of the negative summed squared TSDF values over unprojected depth points, enabling correspondence-free optimization.
• Uses an overlap-aware likelihood to avoid over-aligning non-overlapping regions when fast motion reduces frame overlap.
• Applies PST pre-sampling via 6D Poisson disk sampling and initializes PST for each frame around the previous best pose to maintain exploration.”],
• research_questions_nonstandard_removed? /* placeholder to ensure structure */null? ,
• research_questions: [

Experimental results

Research questions

• RQ1 Can online dense RGB-D reconstruction be robust to very fast camera motion without IMU data?
• RQ2 How can random optimization be made fast and reliable for per-frame pose estimation in depth-based fusion?
• RQ3 Does a pre-sampled particle swarm template improve exploration-exploitation balance in pose optimization?
• RQ4 Is depth-based TSDF conformance sufficient for accurate pose tracking under motion blur?
• RQ5 How does the proposed method perform on fast-motion RGB-D benchmarks compared to existing approaches?

Key findings

• Attains good quality pose tracking under fast camera motion up to 4 m/s at real-time 30 Hz without loop closure or global optimization.
• Demonstrates robustness to motion blur by relying on depth information rather than RGB features.
• Outperforms state-of-the-art methods on fast-motion RGB-D benchmarks, including a new fast-motion benchmark (FastCaMo).
• Provides a depth-only online dense reconstruction system based on PST-evolved random search rather than traditional gradient-based optimization.
• Offers a dataset benchmark for fast-camera-motion RGB-D sequences with ground-truth trajectories and reconstructions.