[Paper Review] GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion

Cheng Chi, Shuran Song|arXiv (Cornell University)|Apr 12, 2021

3D Shape Modeling and Analysis37 references4 citations

TL;DR

GarmentNets proposes a category-level pose estimation method for garments by formulating 3D shape completion in a canonical space shared across garment instances, using a novel winding number field representation to handle thin, non-water-tight structures. It achieves significant performance gains—up to 32.2% improvement over occupancy grids—on shape completion and generalizes effectively to unseen real-world and simulated garments.

ABSTRACT

This paper tackles the task of category-level pose estimation for garments. With a near infinite degree of freedom, a garment's full configuration (i.e., poses) is often described by the per-vertex 3D locations of its entire 3D surface. However, garments are also commonly subject to extreme cases of self-occlusion, especially when folded or crumpled, making it challenging to perceive their full 3D surface. To address these challenges, we propose GarmentNets, where the key idea is to formulate the deformable object pose estimation problem as a shape completion task in the canonical space. This canonical space is defined across garments instances within a category, therefore, specifies the shared category-level pose. By mapping the observed partial surface to the canonical space and completing it in this space, the output representation describes the garment's full configuration using a complete 3D mesh with the per-vertex canonical coordinate label. To properly handle the thin 3D structure presented on garments, we proposed a novel 3D shape representation using the generalized winding number field. Experiments demonstrate that GarmentNets is able to generalize to unseen garment instances and achieve significantly better performance compared to alternative approaches.

Motivation & Objective

• Address the challenge of category-level pose estimation for garments, which have near-infinite degrees of freedom and are prone to severe self-occlusion.
• Overcome the limitations of existing methods that rely on full visibility, known instance meshes, or physics priors, which restrict generalization to unseen garments.
• Develop a robust 3D shape representation suitable for thin, non-water-tight cloth structures that are ill-suited for traditional representations like occupancy grids or signed distance functions.
• Enable generalization to novel garment instances not seen during training using a shared canonical space across a category.
• Support practical robot manipulation by combining partial RGB-D observations with learned shape completion and implicit warp fields to predict accurate poses in task space.

Proposed method

• Define a canonical space for each garment category using a standardized human pose, enabling category-level correspondence learning across diverse garment instances.
• Map partial 3D observations (e.g., RGB-D point clouds) into the canonical space and perform 3D shape completion using a learned neural network to predict a complete, dense 3D mesh.
• Introduce a generalized winding number field (WNF) as a novel 3D shape representation that captures thin, continuous cloth geometry with strong surface gradients and smooth interiors.
• Use a learned implicit warp field network to map the completed canonical mesh back to the real-world task space, correcting errors from earlier stages and improving pose accuracy.
• Train end-to-end using simulated data with pre-aligned 3D meshes and dense correspondence labels, enabling generalization to real-world, unseen garments.
• Compare the learned warp field to physics simulation for back-mapping; the learned field outperforms physics-based methods by self-correcting NOCS prediction errors.

Experimental results

Research questions

• RQ1 Can a canonical space representation enable category-level generalization for garment pose estimation across diverse, unseen garment instances?
• RQ2 How can thin, non-water-tight 3D cloth structures be effectively represented to support accurate shape completion and pose estimation?
• RQ3 Does shape completion in canonical space lead to better generalization and performance compared to direct prediction or physics-based simulation in the task space?
• RQ4 Can a learned implicit warp field outperform physics simulation in mapping completed canonical meshes back to the real-world task space?
• RQ5 To what extent can a model trained on simulated data generalize to real-world RGB-D observations of crumpled, partially visible garments?

Key findings

• GarmentNets achieves a 32.2% improvement in shape completion error over occupancy grids, 18.7% over TDF, and 26.7% over TSDF on the canonical mesh under Chamfer distance.
• The model generalizes effectively to unseen real-world garments, successfully completing 3D geometry and estimating poses from RGB-D data captured with an iPhone 12 Pro Max.
• The learned implicit warp field reduces average pose error by 68.5% compared to physics simulation, with a mean error of 1.41 cm on pants versus 1.64 cm for physics simulation.
• The winding number field representation enables accurate modeling of thin cloth structures, avoiding the over-smoothing of TDF and the voxel-size limitations of occupancy grids.
• On unseen garment instances, the method achieves a mean Chamfer distance of 1.70 cm for shirts and 1.41 cm for pants, demonstrating strong zero-shot generalization.
• The system successfully handles extreme self-occlusion and partial observations by completing the full 3D mesh in canonical space, even when only 30–40% of the surface is visible.