[Paper Review] PAniC-3D: Stylized Single-view 3D Reconstruction from Portraits of Anime Characters

Shuhong Chen, Kevin Zhang+8 more|arXiv (Cornell University)|Mar 25, 2023

Advanced Vision and Imaging4 citations

TL;DR

PAniC-3D proposes a novel system for stylized single-view 3D reconstruction of anime characters from line-based portrait illustrations, using a line-filling model to bridge the illustration-to-3D domain gap and a volumetric radiance field for high-fidelity geometry and texture. It achieves state-of-the-art performance on a new AnimeRecon benchmark, significantly outperforming baselines in both perceptual and geometric fidelity.

ABSTRACT

We propose PAniC-3D, a system to reconstruct stylized 3D character heads directly from illustrated (p)ortraits of (ani)me (c)haracters. Our anime-style domain poses unique challenges to single-view reconstruction; compared to natural images of human heads, character portrait illustrations have hair and accessories with more complex and diverse geometry, and are shaded with non-photorealistic contour lines. In addition, there is a lack of both 3D model and portrait illustration data suitable to train and evaluate this ambiguous stylized reconstruction task. Facing these challenges, our proposed PAniC-3D architecture crosses the illustration-to-3D domain gap with a line-filling model, and represents sophisticated geometries with a volumetric radiance field. We train our system with two large new datasets (11.2k Vroid 3D models, 1k Vtuber portrait illustrations), and evaluate on a novel AnimeRecon benchmark of illustration-to-3D pairs. PAniC-3D significantly outperforms baseline methods, and provides data to establish the task of stylized reconstruction from portrait illustrations.

Motivation & Objective

• To address the challenge of reconstructing stylized 3D anime character heads from single-line portrait illustrations, which are ambiguous and lack photorealistic shading.
• To bridge the domain gap between stylized illustrations and 3D renders, particularly due to non-photorealistic contour lines and complex geometry in hair and accessories.
• To establish a new benchmark and dataset for stylized single-view 3D reconstruction in the anime domain, given the lack of paired illustration-3D data.
• To develop a robust, end-to-end system that generates high-quality, textured 3D radiance fields directly from 2D illustrations without multi-view supervision.
• To enable automated, high-fidelity 3D avatar creation from artist-conceived portraits, reducing reliance on manual 3D modeling.

Proposed method

• The system uses a conditional generative framework based on EG3D, adapted for reconstruction via 2.5D supervision from multi-view 3D renders.
• A novel line-filling model is trained to convert line-based illustrations into clean, render-like images by removing contour lines while preserving facial identity and structure.
• The architecture employs triplane representations with multi-layer feature fusion and feature pooling along triplane axes to improve geometric detail and disambiguation.
• It applies fixed side/back-view supervision signals to enhance geometry consistency and reduce artifacts in occluded regions.
• The model uses a combination of L1, LPIPS, and CLIP-based losses to optimize both pixel-level and semantic-level fidelity.
• The system is trained on a new Vtuber dataset of 1,000 front-facing, neutral-expression portrait illustrations and a Vroid dataset of 11,200 3D character models with multi-view renders.

Figure 1 : Overview of contributions. Our (A) PAniC-3D system is able to reconstruct a 3D radiance field directly from a line-based portrait illustration. We gather a new (B) Vtuber illustration dataset and (C) Vroid 3D models dataset in order to cross the illustration-render domain gap and supervis

Experimental results

Research questions

• RQ1 Can a single-line portrait illustration be used to reconstruct a high-fidelity, stylized 3D radiance field of an anime character?
• RQ2 How can the domain gap between stylized illustrations and 3D renders be effectively bridged to improve reconstruction quality?
• RQ3 What architectural components are most effective for preserving complex geometry and identity in stylized 3D reconstruction?
• RQ4 To what extent can a line-removal model improve downstream 3D reconstruction from illustrations?
• RQ5 How does the proposed system compare quantitatively and qualitatively to existing implicit reconstruction and image-to-image translation baselines?

Key findings

• PAniC-3D achieves a LPIPS score of 18.26 on the AnimeRecon benchmark, significantly outperforming baseline methods such as Telea (23.91) and CycleGAN (21.39).
• The system achieves a CLIP score of 94.97, indicating strong semantic and identity preservation, surpassing UGATIT (85.48) and CycleGAN (93.81).
• The addition of fixed side/back-view supervision improves geometry and texture fidelity, reducing artifacts like extra contours and banding on shoulders.
• Abllation studies confirm that feature pooling and multi-layer triplane features enhance geometric detail and feature disambiguation.
• The line-filling model effectively removes non-photorealistic contours while preserving facial structures, outperforming naive inpainting and off-the-shelf GANs in identity retention.
• Despite strong performance, reconstruction quality still lags behind real 3D assets in occluded regions and ear junctions, indicating room for improvement in object-centric or part-aware generation.

Figure 2 : (a) No-line diffuse render, (b) real-lined illustration, (c) Blender Freestyle [ 14 ] , (d) RTSC suggestive contours [ 9 ] . Toon shaders over-draw (c, cheeks) or miss lines (d, bowtie); it is non-trivial to model the artistic line placement in real drawings (b). Thus, we train our Illust