[论文解读] CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations

Davis Rempe, Tolga Birdal+4 more|arXiv (Cornell University)|Aug 6, 2020

3D Shape Modeling and Analysis参考文献 81被引用 36

一句话总结

CaSPR 使用 Temporal-NOCS 规范化、潜在 ODE 动力学以及 Continuous Normalizing Flow 进行对象中心的规范化时空表示，从而对动态三维点云在不规则数据中实现重建、姿态估计和时空对应关系。

ABSTRACT

We propose CaSPR, a method to learn object-centric Canonical Spatiotemporal Point Cloud Representations of dynamically moving or evolving objects. Our goal is to enable information aggregation over time and the interrogation of object state at any spatiotemporal neighborhood in the past, observed or not. Different from previous work, CaSPR learns representations that support spacetime continuity, are robust to variable and irregularly spacetime-sampled point clouds, and generalize to unseen object instances. Our approach divides the problem into two subtasks. First, we explicitly encode time by mapping an input point cloud sequence to a spatiotemporally-canonicalized object space. We then leverage this canonicalization to learn a spatiotemporal latent representation using neural ordinary differential equations and a generative model of dynamically evolving shapes using continuous normalizing flows. We demonstrate the effectiveness of our method on several applications including shape reconstruction, camera pose estimation, continuous spatiotemporal sequence reconstruction, and correspondence estimation from irregularly or intermittently sampled observations.

研究动机与目标

• Develop an object-centric representation that aggregates spatiotemporal changes in 3D object shapes.
• Canonicalize input dynamic point clouds into a unit-duration spatiotemporal space (T-NOCS).
• Learn a continuous ST latent representation using a Latent ODE and a generative CNF for surface reconstruction.
• Enable reconstruction and querying at arbitrary spatiotemporal resolutions from partial observations.
• Demonstrate applications in shape reconstruction, camera pose estimation, and ST correspondences.

提出的方法

• Canonicalization of 4D ST point clouds to a unit-duration Temporal-NOCS (T-NOCS) via an injective cα(·) mapping.
• A split latent representation zC = [zC ST, zC dyn] with a Latent ODE dz/dt = fθ(zt) to model dynamics in a compact latent space.
• A Continuous Normalizing Flow gβ(·|z) to map Gaussian noise to the object surface at desired timestamps, enabling continuous ST generation.
• Training with a CNF-based log-likelihood loss together with an L1 loss on T-NOCS regression to ground the canonicalization and dynamics.
• Inference where the Latent ODE is solved forward in time for arbitrary canonical timestamps and surfaces are generated via the CNF conditioned on latent states.

实验结果

研究问题

• RQ1 Can dynamic 4D point cloud sequences be canonically normalized to remove extrinsic pose and timing variations?
• RQ2 Can a Latent ODE in a canonical ST space effectively model object dynamics across time?
• RQ3 Can a CNF-based generative model reconstruct continuous spatiotemporal surfaces from partial observations?
• RQ4 Do the learned ST representations support accurate shape reconstruction, pose estimation, and ST correspondence under irregular sampling?
• RQ5 How does CaSPR handle rigid vs non-rigid (deformable) object dynamics and interpolate/unobserve spatiotemporal frames?

主要发现

• CaSPR achieves accurate T-NOCS canonicalization, outperforming several baselines in spatial and temporal alignment for Cars, Chairs, and Airplanes.
• CaSPR provides continuous spatiotemporal reconstruction and maintains temporal continuity better than interpolation-based baselines like PointFlow.
• The SLT (static) and dynamic latent features disentangle shape and motion, enabling plausible motion transfer between sequences.
• CaSPR yields competitive 6D pose estimation accuracy against specialized methods (e.g., RPM-Net) while using canonical T-NOCS points.
• The method supports deformable object reconstruction and maintains correspondences over time better than some baselines for observed and unobserved frames.
• Cross-instance correspondences emerge in the CNF mappings, suggesting potential for label propagation across instances within a category.