[Paper Review] Learning to generate line drawings that convey geometry and semantics

Caroline Chan, Frédo Durand+1 more|arXiv (Cornell University)|Mar 23, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes an unsupervised image-to-image translation method that generates line drawings from photographs by explicitly encoding 3D geometry and semantic content. It introduces a geometry loss based on depth prediction from line drawing features and a semantic loss using CLIP embeddings, outperforming state-of-the-art unpaired methods in conveying shape and identity without paired training data.

ABSTRACT

This paper presents an unpaired method for creating line drawings from photographs. Current methods often rely on high quality paired datasets to generate line drawings. However, these datasets often have limitations due to the subjects of the drawings belonging to a specific domain, or in the amount of data collected. Although recent work in unsupervised image-to-image translation has shown much progress, the latest methods still struggle to generate compelling line drawings. We observe that line drawings are encodings of scene information and seek to convey 3D shape and semantic meaning. We build these observations into a set of objectives and train an image translation to map photographs into line drawings. We introduce a geometry loss which predicts depth information from the image features of a line drawing, and a semantic loss which matches the CLIP features of a line drawing with its corresponding photograph. Our approach outperforms state-of-the-art unpaired image translation and line drawing generation methods on creating line drawings from arbitrary photographs. For code and demo visit our webpage carolineec.github.io/informative_drawings

Motivation & Objective

• To generate line drawings from photographs that effectively convey 3D shape and semantic meaning without requiring paired training data.
• To address the limitations of existing supervised and paired methods, which rely on expensive human-annotated datasets or domain-specific data.
• To improve unsupervised image translation for line drawing generation by incorporating explicit geometry and semantic supervision.
• To reduce dependence on cycle consistency and appearance reconstruction by introducing task-specific losses.
• To enable stylized line drawing generation across diverse subjects and artistic styles using only unpaired images.

Proposed method

• The method formulates line drawing generation as an unsupervised image translation problem using adversarial training and cycle consistency.
• It introduces a geometry loss that predicts depth from features of the generated line drawing, ensuring 3D shape is preserved.
• A semantic loss minimizes the CLIP embedding distance between the input photograph and the generated line drawing to preserve semantic content.
• Appearance preservation is maintained through cycle consistency, ensuring reconstructed images remain faithful to the original.
• The model is trained end-to-end using a combination of adversarial, cycle-consistency, geometry, and semantic losses.
• The approach is evaluated on diverse datasets including portraits, general scenes, and stylized drawings, with ablation studies on loss components.

Figure 2 : Given a photograph $a$ , our model trains network $G_{A}$ to synthesize line drawing $G_{A}(a)$ via four main losses. Adversarial style loss with discriminator $D_{B}$ encourages generated line drawings to match the style of the training set. The CLIP , appearance, and geometry losses enf

Experimental results

Research questions

• RQ1 Can line drawings be effectively generated from unpaired photographs without relying on paired supervision?
• RQ2 Can explicit geometry and semantic supervision improve the quality and informativeness of generated line drawings?
• RQ3 How do depth prediction and CLIP-based semantic alignment contribute to the perceptual quality of line drawings?
• RQ4 To what extent can cycle consistency be reduced when using task-specific losses like geometry and semantics?
• RQ5 How does the method generalize across diverse subjects and artistic styles without fine-tuning?

Key findings

• The method outperforms state-of-the-art unpaired image translation and line drawing generation methods in conveying both geometry and semantics.
• User studies show that participants preferred the proposed method over baselines in 60.1% of cases when trained on the Helen dataset and compared to APDrawingGAN.
• The method achieves strong performance even when trained on general datasets like Helen, surpassing supervised baselines in some settings.
• The semantic loss using CLIP features significantly improves alignment with input semantics, reducing the tendency to produce grayscale-like outputs.
• Ablation studies confirm that both geometry and semantic losses are crucial, with geometry loss improving shape consistency and semantic loss enhancing identity preservation.
• The model generalizes well across styles, producing high-quality results on portraits, complex scenes, and stylized drawings without domain-specific fine-tuning.

Figure 3 : Results of our method in four different styles.