[Paper Review] Learning 3D Dynamic Scene Representations for Robot Manipulation

Zhenjia Xu, Zhanpeng He+2 more|arXiv (Cornell University)|Nov 3, 2020

Human Pose and Action RecognitionComputer Science39 references21 citations

TL;DR

This paper proposes DSR-Net, an end-to-end 3D recurrent neural network that learns a dynamic scene representation (DSR) capturing object permanence, amodal completeness, and spatiotemporal continuity from depth observations. By warping feature maps using predicted scene flow, DSR-Net aggregates visual history spatially coherently, achieving state-of-the-art performance in 3D scene dynamics prediction and enabling accurate model predictive control for robotic manipulation tasks like planar pushing.

ABSTRACT

3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time. In this paper, we introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics while capturing all three properties. We further propose DSR-Net, which learns to aggregate visual observations over multiple interactions to gradually build and refine DSR. Our model achieves state-of-the-art performance in modeling 3D scene dynamics with DSR on both simulated and real data. Combined with model predictive control, DSR-Net enables accurate planning in downstream robotic manipulation tasks such as planar pushing. Video is available at https://youtu.be/GQjYG3nQJ80.

Motivation & Objective

• Address the limitations of 2D and partial 3D visual predictive models that fail under occlusion and in cluttered environments.
• Develop a 3D scene representation that maintains object identity and full geometry even when objects are occluded or partially observed.
• Enable accurate long-horizon prediction of rigid body motion under robot interaction in unstructured, dynamic scenes.
• Integrate the learned representation into model predictive control for improved robotic manipulation planning.
• Establish a benchmark dataset with 80k simulated and 1.5k real-world interactions for evaluating dynamic 3D scene representations.

Proposed method

• Use a 3D volumetric scene encoder to convert depth images into truncated signed distance fields (TSDF) for 3D scene representation.
• Train a motion prediction network to estimate volumetric scene flow from the current scene representation and robot action.
• Apply spatial warping of the scene representation using predicted scene flow to align features across time steps for history aggregation.
• Combine the warped current representation with the next observation’s representation via feature concatenation and 3D convolutions.
• Train the entire DSR-Net framework end-to-end in simulation using a reconstruction and motion prediction loss.
• Fine-tune on real-world data and deploy with model predictive control (MPC) for action planning in planar pushing tasks.

Experimental results

Research questions

• RQ1 Can a 3D scene representation maintain object identity and geometry during occlusion, even when objects are not directly visible?
• RQ2 To what extent does learning spatiotemporal continuity improve long-horizon motion prediction in dynamic, multi-object environments?
• RQ3 How effective is motion-predicted feature warping for aggregating visual history in a spatially coherent manner?
• RQ4 Can a learned 3D dynamic scene representation improve action planning in robotic manipulation tasks such as planar pushing?
• RQ5 How does the proposed method compare to prior state-of-the-art models in both simulated and real-world settings?

Key findings

• DSR-Net achieves a 0.72 IoU in matching target configurations during planar pushing tasks, significantly outperforming SE3-Net (0.31) and SE3Pose-Net (0.32).
• The model successfully maintains object identity and predicts motion during both static and dynamic occlusions, unlike SingleStep and NoWarp baselines.
• DSR-Net achieves high unordered and ordered IoU scores (0.78 and 0.81 in simulation), demonstrating robust spatiotemporal continuity and consistent instance tracking.
• The ablation study confirms that motion-predicted warping improves history aggregation, with GTWarp (ground truth warping) yielding even higher performance.
• The model generalizes to real-world settings, showing consistent performance on a real UR5 robot platform with tabletop planar pushing tasks.
• The proposed benchmark dataset of 80k simulated and 1.5k real-world interactions enables reliable evaluation of dynamic 3D scene representations.