[Paper Review] DART: Articulated Hand Model with Diverse Accessories and Rich Textures

Daiheng Gao, Yuliang Xiu+7 more|arXiv (Cornell University)|Oct 14, 2022

Robot Manipulation and Learning18 citations

TL;DR

DART is a photorealistic, articulated hand model extending MANO with 325 hand-crafted textures and 50 3D accessories to enhance realism. It enables high-fidelity synthetic hand data generation via a Unity GUI, resulting in DARTset—800K diverse, perfectly labeled images that significantly boost generalization in hand pose estimation and mesh recovery tasks.

ABSTRACT

Hand, the bearer of human productivity and intelligence, is receiving much attention due to the recent fever of digital twins. Among different hand morphable models, MANO has been widely used in vision and graphics community. However, MANO disregards textures and accessories, which largely limits its power to synthesize photorealistic hand data. In this paper, we extend MANO with Diverse Accessories and Rich Textures, namely DART. DART is composed of 50 daily 3D accessories which varies in appearance and shape, and 325 hand-crafted 2D texture maps covers different kinds of blemishes or make-ups. Unity GUI is also provided to generate synthetic hand data with user-defined settings, e.g., pose, camera, background, lighting, textures, and accessories. Finally, we release DARTset, which contains large-scale (800K), high-fidelity synthetic hand images, paired with perfect-aligned 3D labels. Experiments demonstrate its superiority in diversity. As a complement to existing hand datasets, DARTset boosts the generalization in both hand pose estimation and mesh recovery tasks. Raw ingredients (textures, accessories), Unity GUI, source code and DARTset are publicly available at dart2022.github.io

Motivation & Objective

• To address the lack of realistic textures and accessories in existing hand models like MANO, which limits photorealistic data synthesis.
• To enable the generation of diverse, high-fidelity synthetic hand images with precise 3D annotations for training vision and graphics models.
• To provide a scalable, interactive pipeline for generating synthetic hand data with user-defined settings such as pose, lighting, background, and accessories.
• To create a large-scale, diverse, and high-fidelity hand dataset (DARTset) that complements existing benchmarks and improves model generalization.
• To release a complete, publicly available toolkit including textures, accessories, source code, GUI, and DARTset for research use.

Proposed method

• Extended MANO with a wrist-enhanced template to support realistic wrist articulation and accessory attachment.
• Designed 325 high-resolution, hand-crafted 2D albedo texture maps covering skin tones, blemishes (moles, scars), make-up (tattoos), and nail colors.
• Created 50 3D textured accessories (rings, watches, bracelets, gloves) with UV maps and mesh models for photorealistic rendering.
• Built a Unity-based GUI and rendering pipeline to generate synthetic images with user-defined parameters: pose, camera, lighting, background, texture, and accessories.
• Generated DARTset—800,000 photorealistic hand images with paired 3D hand meshes, 2D/3D joint annotations, and MANO parameters.
• Used standard metrics (PA-MPJPE, PA-MPVPE) to evaluate performance on pose estimation and mesh reconstruction tasks using cross-dataset training and testing.

Experimental results

Research questions

• RQ1 Can a synthetic hand dataset with rich textures and diverse accessories improve generalization in hand pose estimation and mesh recovery?
• RQ2 How does the inclusion of accessories and realistic textures affect the performance of learning-based hand reconstruction models?
• RQ3 To what extent does DARTset reduce domain gap when used in combination with real-world datasets like FreiHAND?
• RQ4 Can the DART framework and DARTset be effectively used to train models that generalize to in-the-wild hand images?
• RQ5 How does the diversity of poses and textures in DARTset compare to existing benchmarks in terms of model generalization and robustness?

Key findings

• DARTset significantly improves generalization in hand pose estimation and mesh recovery, with a 7.8% relative improvement in PA-MPJPE for Integral Pose and 5.9% for CMR when using accessories.
• Cross-dataset training with DARTset and FreiHAND reduced PA-MPVPE by 8.9% on FreiHAND for CMR, demonstrating DARTset's ability to complement real-world datasets.
• The CMR model achieved 4.84/3.46 cm PA-MPJPE/PA-MPVPE on DARTset when trained on mixed data, showing strong performance on the synthetic set.
• METRO achieved 3.82/3.73 cm PA-MPJPE/PA-MPVPE on FreiHAND when trained on mixed data, indicating DARTset's effectiveness in domain generalization.
• The ablation study confirmed that accessories alone improve PA-MPJPE by 7.8% for Integral Pose and 5.9% for CMR, proving their critical role in realism and performance.
• DARTset’s large-scale, diverse pose distribution and high-fidelity textures enable better generalization than existing datasets, especially when combined with real-world data.