[Paper Review] DeepWriting: Making Digital Ink Editable via Deep Generative Modeling

Emre Aksan, Fabrizio Pece+1 more|arXiv (Cornell University)|Jan 25, 2018

Handwritten Text Recognition Techniques10 references8 citations

TL;DR

This paper introduces DeepWriting, a deep generative model that disentangles handwriting style from content using a conditional variational recurrent neural network (C-VRNN), enabling editable digital ink through style transfer, word-level editing, and text-to-handwriting synthesis. The key contribution is a learnable latent representation that decouples style and content, allowing full control over synthesis and editing while preserving individual writing characteristics.

ABSTRACT

Digital ink promises to combine the flexibility and aesthetics of handwriting and the ability to process, search and edit digital text. Character recognition converts handwritten text into a digital representation, albeit at the cost of losing personalized appearance due to the technical difficulties of separating the interwoven components of content and style. In this paper, we propose a novel generative neural network architecture that is capable of disentangling style from content and thus making digital ink editable. Our model can synthesize arbitrary text, while giving users control over the visual appearance (style). For example, allowing for style transfer without changing the content, editing of digital ink at the word level and other application scenarios such as spell-checking and correction of handwritten text. We furthermore contribute a new dataset of handwritten text with fine-grained annotations at the character level and report results from an initial user evaluation.

Motivation & Objective

• To address the challenge of making digital ink editable while preserving personalized handwriting style.
• To develop a deep generative model capable of disentangling content and style in handwriting for flexible editing and synthesis.
• To enable applications such as spell-checking, auto-correction, and style transfer on handwritten digital ink.
• To collect and release a large-scale, character-annotated dataset of handwritten text from 294 authors to support future research.
• To provide an open-source implementation of the model for community adoption and extension.

Proposed method

• The method employs a conditional variational recurrent neural network (C-VRNN) that models content and style as separate latent variables with distinct distributions.
• Content is modeled via a recurrent variational autoencoder (RVAE) that encodes character sequences into a latent space.
• Style is modeled through a separate latent distribution that captures user-specific handwriting characteristics such as slant, size, and pressure.
• The model uses a novel training and sampling algorithm to enable conditional generation by controlling both content and style latents independently.
• The architecture supports conditional synthesis: generating ink from typed text, transferring style between writers, and editing individual words.
• A character-level segmentation pipeline was applied to raw stylus data to produce fine-grained annotations for training and evaluation.

Experimental results

Research questions

• RQ1 Can a deep generative model effectively disentangle handwriting style from content to enable editable digital ink?
• RQ2 How well can the model transfer styles between different writers while preserving content accuracy?
• RQ3 To what extent can users edit handwritten text at the word level using the proposed framework?
• RQ4 How does the model perform in generating both cursive and disconnected handwriting styles?
• RQ5 What is the impact of fine-grained character-level annotations on model performance and downstream applications?

Key findings

• The model successfully disentangles content and style, enabling conditional synthesis of digital ink from typed text with user-defined visual appearance.
• The system supports style transfer between handwriting samples without altering content, as demonstrated in user evaluation scenarios.
• Word-level editing of digital ink is feasible and perceived positively by users in preliminary evaluations.
• The model achieves higher fidelity in generating disconnected (non-cursive) handwriting compared to cursive styles, which remains a current limitation.
• The proposed dataset of 294 authors with character-level annotations enhances the IAM-OnDB and enables robust training and evaluation of handwriting models.
• An open-source implementation of the model and dataset is publicly released to support reproducibility and further research.