[Paper Review] Deep Generation of Face Images from Sketches

Shuyu Chen, Wanchao Su+3 more|arXiv (Cornell University)|Jun 1, 2020

Generative Adversarial Networks and Image Synthesis41 references4 citations

TL;DR

This paper proposes DeepFaceDrawing, a deep learning framework that generates high-quality, realistic face images from rough or incomplete freehand sketches by implicitly modeling face component manifolds (eyes, nose, mouth) in a feature space. Using a local-to-global approach with component embedding, feature mapping, and image synthesis modules, the system treats sketches as soft constraints, enabling robust synthesis even from low-quality inputs while preserving user intent and supporting fine-grained control.

ABSTRACT

Recent deep image-to-image translation techniques allow fast generation of face images from freehand sketches. However, existing solutions tend to overfit to sketches, thus requiring professional sketches or even edge maps as input. To address this issue, our key idea is to implicitly model the shape space of plausible face images and synthesize a face image in this space to approximate an input sketch. We take a local-to-global approach. We first learn feature embeddings of key face components, and push corresponding parts of input sketches towards underlying component manifolds defined by the feature vectors of face component samples. We also propose another deep neural network to learn the mapping from the embedded component features to realistic images with multi-channel feature maps as intermediate results to improve the information flow. Our method essentially uses input sketches as soft constraints and is thus able to produce high-quality face images even from rough and/or incomplete sketches. Our tool is easy to use even for non-artists, while still supporting fine-grained control of shape details. Both qualitative and quantitative evaluations show the superior generation ability of our system to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.

Motivation & Objective

• To address the limitation of existing sketch-to-image models that require high-quality or edge-map-like sketches for realistic face generation.
• To enable high-quality face image synthesis from rough, incomplete, or low-skill freehand sketches.
• To model plausible face component shapes implicitly via low-dimensional manifolds in feature space for improved robustness and user-friendliness.
• To support fine-grained control over facial component shapes while maintaining global facial coherence.
• To evaluate usability and expressiveness through user studies and real-world applications like face morphing and copy-paste.

Proposed method

• The system decomposes input sketches into five components: left-eye, right-eye, nose, mouth, and remainder, each processed independently.
• A component embedding (CE) module uses an auto-encoder to learn feature descriptors and project sketch components onto underlying component manifolds in latent space.
• A feature mapping (FM) module maps the embedded component features to 32-channel feature maps, improving information flow and enabling spatial fusion.
• An image synthesis (IS) module generates the final 512×512 face image from the fused feature maps using a conditional GAN-like architecture.
• A shadow-guided interface assists users by projecting sketch components to component manifolds, enabling easier sketching with structural guidance.
• The framework treats input sketches as soft constraints, allowing faithful reconstruction of user intent even with incomplete or imprecise strokes.

Experimental results

Research questions

• RQ1 Can a deep learning model generate realistic face images from rough or incomplete freehand sketches without requiring professional-quality input?
• RQ2 How can component-level feature manifolds be implicitly modeled to improve robustness to sketch quality?
• RQ3 To what extent can a local-to-global approach with component-wise refinement enhance synthesis quality and user control?
• RQ4 How does the system maintain global facial coherence while allowing local editing of facial components?
• RQ5 Can the proposed method outperform existing sketch-to-image translation models in both qualitative and quantitative evaluations?

Key findings

• The proposed method generates high-quality, realistic face images (512×512 resolution) even from rough or incomplete sketches, outperforming existing methods in visual quality.
• User studies confirmed the system’s usability and expressiveness, showing that non-artists can produce plausible face images with minimal drawing skill.
• The system preserves user intent in input sketches, treating them as soft constraints rather than hard constraints, which improves robustness to sketch imperfections.
• Quantitative and qualitative evaluations demonstrate superior performance compared to baseline and alternative sketch-to-image models.
• The method supports local editing and fine-grained control of facial component shapes, enabling applications such as face morphing and copy-paste.
• Despite limitations in handling layout errors or color control, the system shows strong potential for adaptation to other object categories with structural priors.