[Paper Review] IKEA-Manual: Seeing Shape Assembly Step by Step

Ruocheng Wang, Yunzhi Zhang+4 more|arXiv (Cornell University)|Feb 3, 2023

3D Surveying and Cultural Heritage5 citations

TL;DR

This paper introduces IKEA-Manual, a new dataset of 102 real-world IKEA furniture 3D models paired with human-designed visual assembly manuals, featuring fine-grained annotations including part decomposition, assembly plans, manual segmentation, and 2D-3D correspondences. The dataset enables structured understanding of shape assembly and demonstrates utility in four tasks: assembly plan generation, part segmentation, pose estimation, and 3D part assembly, with significant performance drops on real-world parts indicating a need for improved generalization beyond synthetic benchmarks.

ABSTRACT

Human-designed visual manuals are crucial components in shape assembly activities. They provide step-by-step guidance on how we should move and connect different parts in a convenient and physically-realizable way. While there has been an ongoing effort in building agents that perform assembly tasks, the information in human-design manuals has been largely overlooked. We identify that this is due to 1) a lack of realistic 3D assembly objects that have paired manuals and 2) the difficulty of extracting structured information from purely image-based manuals. Motivated by this observation, we present IKEA-Manual, a dataset consisting of 102 IKEA objects paired with assembly manuals. We provide fine-grained annotations on the IKEA objects and assembly manuals, including decomposed assembly parts, assembly plans, manual segmentation, and 2D-3D correspondence between 3D parts and visual manuals. We illustrate the broad application of our dataset on four tasks related to shape assembly: assembly plan generation, part segmentation, pose estimation, and 3D part assembly.

Motivation & Objective

• To address the lack of realistic 3D assembly objects paired with human-designed visual manuals in existing datasets.
• To bridge the gap between human-readable visual manuals and machine-interpretable assembly processes by structuring unstructured manual images.
• To enable new research directions in shape assembly by providing dense, multi-level annotations linking 3D parts to 2D manual projections.
• To support the development and benchmarking of models for assembly planning, part segmentation, pose estimation, and 3D part assembly.
• To highlight the limitations of current models when generalizing from synthetic benchmarks (e.g., PartNet) to real-world part decompositions.

Proposed method

• The dataset collects 102 3D IKEA furniture models from existing repositories and pairs them with official assembly manuals from IKEA’s website.
• Each 3D model is decomposed into primitive parts that match the manual’s step-by-step instructions, with connection relationships annotated.
• Assembly manuals are parsed into structured formats by manually identifying connected parts per step and segmenting 2D projections of 3D parts in manual images.
• Keypoints are annotated on 3D parts and their 2D projections to compute 3D poses and establish 2D-3D correspondences.
• The dataset supports four tasks: assembly plan generation, part segmentation, pose estimation, and 3D part assembly, using the same annotated data for evaluation.
• Baselines from prior work (e.g., RGL-Net, Huang et al.) are evaluated on the dataset to assess generalization performance on real-world parts.

Experimental results

Research questions

• RQ1 Can a dataset with real-world 3D furniture models and paired human-designed visual manuals enable more realistic and structured shape assembly learning?
• RQ2 How well do existing models for part assembly generalize when evaluated on real-world part decompositions versus synthetic benchmarks?
• RQ3 To what extent can fine-grained 2D-3D correspondences in visual manuals improve part segmentation and pose estimation?
• RQ4 Can the structured assembly plans derived from manuals be used to train models for high-level assembly planning?
• RQ5 What are the limitations of current models when applied to real-world part decompositions, and how do they differ from synthetic benchmarks?

Key findings

• The performance of existing part assembly models degrades significantly when evaluated on IKEA-Manual compared to PartNet, with a 75% drop in part accuracy from 49.06% to 3.99% for RGL-Net*.
• The shape Chamfer distance (CD) increases from 0.0087 on PartNet to 0.0508 on IKEA-Manual, indicating poorer reconstruction quality on real-world parts.
• Part accuracy drops more sharply than shape CD, suggesting models struggle with semantic part identification despite forming plausible overall shapes.
• The baseline model Huang et al. [4] achieves 6.90% part accuracy on IKEA-Manual, significantly lower than its 39.96% on PartNet, highlighting generalization failure.
• The dataset reveals that simplified, synthetic parts in benchmarks like PartNet do not capture the complexity and variability of real-world parts, limiting model robustness.
• The results demonstrate that current models trained on synthetic data fail to generalize to real-world part decompositions, underscoring the need for datasets like IKEA-Manual to close this gap.