[Paper Review] Learning to Sketch with Shortcut Cycle Consistency

Jifei Song, Kaiyue Pang+3 more|arXiv (Cornell University)|May 1, 2018

Advanced Image and Video Retrieval Techniques42 references15 citations

TL;DR

This paper proposes a novel deep learning framework for stroke-level photo-to-sketch synthesis that leverages shortcut cycle consistency to improve robustness under noisy, weak supervision from human sketch pairs. By combining supervised translation with unsupervised within-domain reconstruction and inverse-domain reconstruction, the model generates semantically meaningful, abstract sketches that outperform state-of-the-art methods in both recognition and retrieval benchmarks, while also enabling effective data augmentation for fine-grained sketch-based image retrieval.

ABSTRACT

To see is to sketch -- free-hand sketching naturally builds ties between human and machine vision. In this paper, we present a novel approach for translating an object photo to a sketch, mimicking the human sketching process. This is an extremely challenging task because the photo and sketch domains differ significantly. Furthermore, human sketches exhibit various levels of sophistication and abstraction even when depicting the same object instance in a reference photo. This means that even if photo-sketch pairs are available, they only provide weak supervision signal to learn a translation model. Compared with existing supervised approaches that solve the problem of D(E(photo)) -> sketch, where E($\cdot$) and D($\cdot$) denote encoder and decoder respectively, we take advantage of the inverse problem (e.g., D(E(sketch)) -> photo), and combine with the unsupervised learning tasks of within-domain reconstruction, all within a multi-task learning framework. Compared with existing unsupervised approaches based on cycle consistency (i.e., D(E(D(E(photo)))) -> photo), we introduce a shortcut consistency enforced at the encoder bottleneck (e.g., D(E(photo)) -> photo) to exploit the additional self-supervision. Both qualitative and quantitative results show that the proposed model is superior to a number of state-of-the-art alternatives. We also show that the synthetic sketches can be used to train a better fine-grained sketch-based image retrieval (FG-SBIR) model, effectively alleviating the problem of sketch data scarcity.

Motivation & Objective

• To address the challenge of learning photo-to-sketch translation under highly noisy supervision from human sketch pairs, which vary widely in abstraction and style.
• To improve sketch generation quality by combining supervised learning with unsupervised self-supervision through within-domain reconstruction and inverse-domain translation.
• To develop a stroke-level generative model that mimics human sketching behavior, producing abstract yet semantically faithful depictions of objects.
• To evaluate whether synthetic sketches can effectively augment real photo-sketch pairs for training better fine-grained sketch-based image retrieval (FG-SBIR) models.

Proposed method

• The model employs an encoder-decoder architecture with shared parameters across multiple tasks to enable multi-task learning.
• It introduces shortcut cycle consistency by enforcing reconstruction of the input photo through the encoder and decoder without leaving the photo domain, i.e., D(E(photo)) → photo.
• The framework integrates three main tasks: (1) supervised photo-to-sketch translation D(E(photo)) → sketch, (2) inverse sketch-to-photo translation D(E(sketch)) → photo, and (3) within-domain reconstruction for both domains.
• A variational autoencoder-style KL loss is applied to the photo encoder to enable disentangled and diverse sampling in the latent space for generating multiple distinct sketches from the same input.
• The model is trained end-to-end using a multi-task loss combining reconstruction, cycle consistency, and adversarial objectives.
• Synthetic sketches are generated from unseen test photos and used to pre-train an FG-SBIR model, improving downstream performance.

Experimental results

Research questions

• RQ1 Can a deep learning model generate high-quality, abstract, and semantically meaningful sketches from a single photo, even with highly variable human sketch supervision?
• RQ2 Does incorporating shortcut cycle consistency—reconstructing inputs within the same domain—improve sketch generation compared to full cycle consistency?
• RQ3 Can synthetic sketches generated by the model effectively serve as data augmentation to improve fine-grained sketch-based image retrieval (FG-SBIR) performance?
• RQ4 How does the model’s performance compare to human sketches and state-of-the-art models in terms of recognition and retrieval accuracy?

Key findings

• On the ShoeV2 dataset, the proposed model achieves 53.50% top-1 accuracy in sketch recognition and 6.00% top-1 accuracy in sketch-based image retrieval, outperforming all competitors, including human sketches in recognition.
• On the smaller ChairV2 dataset, the model achieves 13.00% top-1 accuracy in recognition and 8.00% in retrieval, surpassing even the strong CycleGAN-S baseline.
• The model’s performance is significantly better than chance (0.5% acc.@1 on ShoeV2 and 1% on ChairV2), indicating that synthesized sketches retain instance-identifiable details.
• When used for data augmentation in FG-SBIR, synthetic sketches improved the model’s top-1 accuracy by 2.10 percentage points (from 30.33% to 32.43%).
• Latent space sampling produces diverse, globally consistent sketches from the same input, demonstrating the model’s ability to generate multiple plausible interpretations of the same object.
• The model generates sketches that are semantically abstract and perceptually distinct from simple rasterized copies, as evidenced by stroke-level rendering of parts like shoelaces and chair legs.