[Paper Review] Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches

Weihao Xia, Yujiu Yang+1 more|arXiv (Cornell University)|Nov 1, 2019

Generative Adversarial Networks and Image Synthesis64 references11 citations

TL;DR

Cali-Sketch proposes a two-stage method for generating photo-realistic face images from poorly-drawn sketches by decoupling stroke calibration and image synthesis. It uses a Stroke Calibration Network to refine facial strokes and enrich details while preserving intent, followed by an Image Synthesis Network to generate high-fidelity photos, outperforming state-of-the-art methods in realism and fidelity.

ABSTRACT

Image generation task has received increasing attention because of its wide application in security and entertainment. Sketch-based face generation brings more fun and better quality of image generation due to supervised interaction. However, When a sketch poorly aligned with the true face is given as input, existing supervised image-to-image translation methods often cannot generate acceptable photo-realistic face images. To address this problem, in this paper we propose Cali-Sketch, a poorly-drawn-sketch to photo-realistic-image generation method. Cali-Sketch explicitly models stroke calibration and image generation using two constituent networks: a Stroke Calibration Network (SCN), which calibrates strokes of facial features and enriches facial details while preserving the original intent features; and an Image Synthesis Network (ISN), which translates the calibrated and enriched sketches to photo-realistic face images. In this way, we manage to decouple a difficult cross-domain translation problem into two easier steps. Extensive experiments verify that the face photos generated by Cali-Sketch are both photo-realistic and faithful to the input sketches, compared with state-of-the-art methods

Motivation & Objective

• To address the challenge of generating photo-realistic face images from poorly-drawn, misaligned sketches.
• To improve image quality and fidelity when input sketches deviate significantly from true facial structures.
• To decouple the complex image-to-image translation task into two manageable stages: stroke calibration and image synthesis.
• To preserve the original sketch's intent while enhancing facial details and alignment.

Proposed method

• A Stroke Calibration Network (SCN) is introduced to correct misaligned strokes and enrich facial details in poorly-drawn sketches.
• The SCN preserves the original sketch's structural intent while refining stroke geometry and adding fine-grained facial features.
• An Image Synthesis Network (ISN) translates the calibrated sketches into photo-realistic face images.
• The two networks are trained sequentially, with the SCN refining input sketches before the ISN generates the final image.
• The method decouples the difficult cross-domain translation into two easier sub-tasks, improving training stability and output quality.
• The framework enables better alignment between generated images and the original sketch's intended facial structure.

Experimental results

Research questions

• RQ1 Can stroke calibration improve the quality of facial features in poorly-drawn sketches before image generation?
• RQ2 How effectively can a two-stage network architecture enhance photo-realism and fidelity compared to end-to-end methods?
• RQ3 To what extent does preserving the original sketch's intent improve the realism of generated face images?
• RQ4 Does decoupling stroke calibration from image synthesis lead to better performance than joint optimization?

Key findings

• Cali-Sketch generates photo-realistic face images that are both visually realistic and faithful to the input sketch's original intent.
• The stroke calibration step significantly improves facial feature alignment and detail enrichment in low-quality sketches.
• The two-stage approach outperforms state-of-the-art methods in generating high-fidelity face images from poorly-drawn inputs.
• Extensive experiments confirm that Cali-Sketch maintains structural consistency with the input sketch while producing realistic textures and facial details.