[Paper Review] Generative 3D Part Assembly via Dynamic Graph Learning

Jialei Huang, Guanqi Zhan+6 more|arXiv (Cornell University)|Jun 14, 2020

3D Shape Modeling and AnalysisEngineering40 references46 citations

TL;DR

The paper proposes an assembly-oriented dynamic graph learning framework that uses an iterative GNN with dynamic relation reasoning and part aggregation to predict 6-DoF poses for input 3D parts, achieving state-of-the-art results on PartNet chairs, tables, and lamps.

ABSTRACT

Autonomous part assembly is a challenging yet crucial task in 3D computer vision and robotics. Analogous to buying an IKEA furniture, given a set of 3D parts that can assemble a single shape, an intelligent agent needs to perceive the 3D part geometry, reason to propose pose estimations for the input parts, and finally call robotic planning and control routines for actuation. In this paper, we focus on the pose estimation subproblem from the vision side involving geometric and relational reasoning over the input part geometry. Essentially, the task of generative 3D part assembly is to predict a 6-DoF part pose, including a rigid rotation and translation, for each input part that assembles a single 3D shape as the final output. To tackle this problem, we propose an assembly-oriented dynamic graph learning framework that leverages an iterative graph neural network as a backbone. It explicitly conducts sequential part assembly refinements in a coarse-to-fine manner, exploits a pair of part relation reasoning module and part aggregation module for dynamically adjusting both part features and their relations in the part graph. We conduct extensive experiments and quantitative comparisons to three strong baseline methods, demonstrating the effectiveness of the proposed approach.

Motivation & Objective

• Motivate autonomous 3D part assembly by predicting per-part 6-DoF poses from input part geometry without semantic priors.
• Develop an assembly-oriented dynamic graph framework that evolves part features and relations for coarse-to-fine pose refinement.
• Leverage dynamic relation reasoning and dynamic part aggregation to improve pose estimation and final shape connectivity.
• Evaluate on synthetic PartNet data and compare against strong baselines to demonstrate effectiveness.

Proposed method

• Represent parts as nodes in a dynamic, fully connected graph at each iteration.
• Encode each part with a PointNet-based feature from its point cloud as initial node features.
• Perform 5 iterations of graph message passing with time-varying edge and node updates to refine poses.
• Introduce a dynamic relation reasoning module that learns directed edge weights r_ij to modulate message passing based on current pose estimates.
• Introduce a dynamic part aggregation module that groups geometrically-equivalent parts into sparse nodes to propagate shared information and then unpool back to dense nodes.
• Train with a Min-of-N (MoN) loss to handle multimodal assembly outputs, supervising rotation and translation per part and overall shape via Chamfer-based losses.
• Use PartNet data (chairs, tables, lamps) with synthetic ground-truth part poses for supervised learning and evaluation.

Experimental results

Research questions

• RQ1 Can an assembly-oriented dynamic graph model learn coherent, sequential pose refinements for multi-part 3D shapes without semantic priors?
• RQ2 Do dynamic relation reasoning and dynamic part aggregation improve the accuracy and connectivity of assembled shapes compared to static or non-graph baselines?
• RQ3 How does the model perform across fine-grained parts in furniture-like shapes on the PartNet dataset?
• RQ4 What insights exist about the iterative refinement dynamics of central versus peripheral parts during assembly?

Key findings

• The proposed method outperforms three strong baselines on Shape Chamfer Distance, Part Accuracy, and Connectivity Accuracy across Chair/Table/Lamp categories.
• Central parts (e.g., chair back/seat) are learned to pose earlier and guide peripheral parts (legs/arms) in subsequent iterations, as shown by dynamic relation weights.
• Dynamic relation reasoning and dynamic part aggregation each contribute to performance, with ablations showing notable drops when removed.
• The full model achieves the best results: Shape CD 0.0050, Part Accuracy 49.51 (Chair), 33.33 (Lamp/Table), Connectivity 39.96 (Chair), 41.70 (Lamp).
• Ablation studies indicate that 5 GNN iterations provide a good balance between accuracy and trainability; more iterations offer diminishing returns.