[Paper Review] DrawTalking: Building Interactive Worlds by Sketching and Speaking

Karl Rosenberg, Rubaiat Habib Kazi+3 more|arXiv (Cornell University)|Jan 11, 2024

Art, Technology, and Culture5 citations

TL;DR

DrawTalking presents a multimodal prototype that lets users build and control interactive worlds by sketching and speaking, without coding.

ABSTRACT

We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

Motivation & Objective

• Enable open-ended creative exploration by letting users sketch and speak to define content and behavior.
• Provide semantic labeling of sketches via speech to control objects and their properties.
• Balance user agency with automated system assistance to support rapid prototyping and iteration.
• Offer a transparent interface that visualizes speech input and system understanding to support error correction.
• Demonstrate applicability to educational, rapid prototyping, and ideation workflows.

Proposed method

• Develop a multitouch iPad app where freehand sketches are labeled with nouns, adjectives, and adverbs through speech and touch.
• Use deixis and text labeling to connect sketches with semantic roles for control and behavior.
• Implement a language command system with primitives (verbs, conjunctions, prepositions) to drive animations and events.
• Provide a semantics diagram and transcript view to visualize and edit the system’s understanding of commands.
• Enable runtime editing and immediate playtesting through direct manipulation and scripted rules.
• Implement on-device speech recognition with a local NLP processing pipeline to map language to actions.

Experimental results

Research questions

• RQ1 How understandable and usable is the sketch+speech interface for building interactive scenes?
• RQ2 What kinds of use cases and workflows do participants identify for open-ended creative exploration?
• RQ3 How do users label and command sketches, and how effective is the semantics diagram for error recovery?
• RQ4 Can users perform programming-like tasks (loops, rules, spawning) without writing code?
• RQ5 What are participants’ perceptions of DrawTalking’s potential in education, rapid prototyping, and creative pipelines.

Key findings

• Participants understood and learned the mechanics and engaged in thoughtful ideation.
• Labeling via deixis (speech-driven labeling) was preferred for intuitiveness over linking sketches to words.
• The semantics diagram helped users inspect and correct the system’s interpretation of commands.
• Users reported a sense of physicality and described DrawTalking as approachable, akin to “language and Legos”.
• Participants saw potential for integration with other tools and for rapid prototyping in games and education.
• Some users viewed DrawTalking as a comparative stepping-stone toward programming environments while remaining accessible to non-programmers.