[Paper Review] Deep Learning for Identifying Potential Conceptual Shifts for Co-creative Drawing

Pegah Karimi, Nicholas M. Davis+2 more|arXiv (Cornell University)|Jan 2, 2018

Design Education and Practice8 references3 citations

TL;DR

This paper presents a co-creative drawing system that uses deep learning to identify conceptual shifts by matching user sketches to structurally similar sketches from different categories. Trained on the Quick, Draw! dataset using VGG-16 features and k-means clustering, the system detects potential conceptual blends through Euclidean distance between cluster centroids, enabling AI to suggest creative, cross-category sketch transformations to enhance user creativity.

ABSTRACT

We present a system for identifying conceptual shifts between visual categories, which will form the basis for a co-creative drawing system to help users draw more creative sketches. The system recognizes human sketches and matches them to structurally similar sketches from categories to which they do not belong. This would allow a co-creative drawing system to produce an ambiguous sketch that blends features from both categories.

Motivation & Objective

• To develop a co-creative drawing system that enhances user creativity through AI-driven conceptual shifts.
• To address the challenge of generating novel sketch ideas by identifying structural similarities across different visual categories.
• To enable free association between unrelated sketch categories by detecting potential conceptual blends.
• To train a system that can recognize user sketches and respond with structurally similar but category-different sketches.

Proposed method

• The system uses a pre-trained VGG-16 network fine-tuned on the Quick, Draw! dataset to extract 4096-dimensional feature vectors from sketches.
• Sketches are embedded into high-level representations using the first fully connected layer (fc1) of VGG-16.
• k-means clustering is applied to these embeddings to group structurally similar sketches into sub-categories within each object category.
• Conceptual shifts are identified by computing Euclidean distances between the centroid of the user’s sketch cluster and centroids of clusters in other categories.
• The closest cluster from a different category is selected as the most promising conceptual shift.
• The system contributes a response sketch from the matched cluster, currently via random selection, with future plans for neural generation.

Experimental results

Research questions

• RQ1 How can a co-creative system detect structural similarities between sketches from different conceptual categories to enable conceptual blending?
• RQ2 What deep learning-based feature representation enables effective clustering of diverse sketch variations within the same category?
• RQ3 How can clustering on deep features improve the identification of potential conceptual shifts in free association sketching?
• RQ4 What metrics or criteria can reliably rank the most promising conceptual shifts from different categories?
• RQ5 How does the system’s response vary across multiple inputs and iterations, and can it generate diverse, contextually relevant conceptual shifts?

Key findings

• The system successfully identifies structurally similar sketches across different categories using Euclidean distance between cluster centroids, enabling potential conceptual shifts.
• Clustering on VGG-16 features effectively groups diverse sketch variations of the same object into meaningful sub-categories, reducing intra-category variance.
• Case studies demonstrate that the system can generate multiple plausible conceptual shift responses for a single input sketch, such as transforming a potato into a shape resembling a flower or a tooth.
• The system produces visually plausible conceptual blends, with low-distance matches indicating strong structural similarity, as shown in Figure 3.
• For multiple inputs of the same object (e.g., different flowers or teeth), the system identifies distinct conceptual shifts, indicating adaptability to evolving user input.
• The approach enables a free association game framework where AI and user co-create by exploring cross-category visual transformations.