[Paper Review] 3D Reconstruction of Deformable Revolving Object under Heavy Hand Interaction

Raoul de Charette, Sotiris Manitsaris|arXiv (Cornell University)|Aug 5, 2019

Optical measurement and interference techniques30 references4 citations

TL;DR

This paper proposes a particle energy optimization method for 3D reconstruction of deformable, radially symmetric objects—such as clay in pottery—under heavy hand interaction. Using one or more depth sensors, it leverages radial symmetry and temporal consistency to achieve robust reconstruction despite occlusion and deformation, attaining a state-of-the-art 7.60 mm average reconstruction error.

ABSTRACT

We reconstruct 3D deformable object through time, in the context of a live pottery making process where the crafter molds the object. Because the object suffers from heavy hand interaction, and is being deformed, classical techniques cannot be applied. We use particle energy optimization to estimate the object profile and benefit of the object radial symmetry to increase the robustness of the reconstruction to both occlusion and noise. Our method works with an unconstrained scalable setup with one or more depth sensors. We evaluate on our database (released upon publication) on a per-frame and temporal basis and shows it significantly outperforms state-of-the-art achieving 7.60mm average object reconstruction error. Further ablation studies demonstrate the effectiveness of our method.

Motivation & Objective

• Address the challenge of 3D reconstruction of unknown, deformable, radially symmetric objects during interactive manipulation, such as in pottery making.
• Overcome limitations of existing methods that rely on known templates, rigid assumptions, or texture priors, which fail under heavy occlusion and deformation.
• Enable robust reconstruction in unconstrained, scalable setups with one or more depth sensors, without requiring prior knowledge of object shape or texture.
• Ensure high accuracy despite significant partial occlusion from hands and dynamic shape evolution during the modeling process.
• Provide a method that is both temporally consistent and resilient to noise and sensor variability through radial symmetry exploitation and particle-based optimization.

Proposed method

• Use radial symmetry of the object to map 3D point clouds into a 2D polar (ρ,h) space, reducing the problem to profile estimation.
• Apply particle energy optimization in the polar domain to estimate the object’s 3D profile by minimizing a combination of geometric and appearance-based energy terms.
• Employ a radial accumulator to discretize the polar space, with resolution tuned to balance accuracy and computational cost.
• Integrate multiple depth sensors by registering their point clouds and fusing them into a unified input for the reconstruction pipeline.
• Use temporal filtering and particle resampling (with 0.8 weight) to improve stability and reduce jitter across frames.
• Leverage the object’s radial symmetry to enhance robustness to partial occlusion and noise, especially when hands cover parts of the object.

Experimental results

Research questions

• RQ1 Can 3D reconstruction of unknown, deformable, revolving objects be achieved under heavy hand interaction without relying on prior shape or texture models?
• RQ2 How does radial symmetry improve reconstruction robustness under partial occlusion and sensor noise in dynamic, interactive scenarios?
• RQ3 What is the impact of sensor count and particle count on reconstruction accuracy and computational efficiency?
• RQ4 How does the resolution of the radial accumulator affect reconstruction precision and processing time?
• RQ5 Can a scalable, unconstrained depth sensor setup achieve sub-centimeter reconstruction accuracy in real-time interactive applications?

Key findings

• The proposed method achieves a mean reconstruction error of 7.60 mm, significantly outperforming state-of-the-art methods, which report errors above 10 mm under similar conditions.
• Using two depth sensors reduces the average error by 17.20% compared to using only the best single sensor (8.09 mm vs. 9.77 mm), demonstrating sensor fusion benefits.
• Increasing particle count from 100 to 1000 improves accuracy by 19.74%, but further gains beyond 1000 particles are marginal (only 4.60% improvement at 5000 particles).
• Higher radial accumulator resolution (64x64) reduces average error by 11.39% compared to 16x16 (8.43 mm vs. 9.39 mm), though processing time increases by 5.35×.
• The method remains robust to sensor ablation and partial occlusion, maintaining sub-10 mm error even with one sensor or high noise levels.
• Despite improvements, maximum error (HD) remains above 20 mm in some cases, indicating limitations under severe occlusion, suggesting a need for spatio-temporal priors for complete occlusion handling.