[Paper Review] Neural Trajectory Analysis of Recurrent Neural Network In Handwriting Synthesis

Kristof B. Charbonneau, Osamu Shouno|arXiv (Cornell University)|Apr 13, 2018

Handwritten Text Recognition Techniques2 references3 citations

TL;DR

This paper introduces neural trajectory analysis to interpret how recurrent neural networks (RNNs) generate diverse handwriting styles. By applying Gaussian-Process Factor Analysis (GPFA) to LSTM activations, the study reveals that different writing styles are encoded in distinct subspaces of the network's internal space, while individual characters are represented as unique state dynamics within each style's subspace—highlighting the role of specific recurrent layers in style retention and character generation.

ABSTRACT

Recurrent neural networks (RNNs) are capable of learning to generate highly realistic, online handwritings in a wide variety of styles from a given text sequence. Furthermore, the networks can generate handwritings in the style of a particular writer when the network states are primed with a real sequence of pen movements from the writer. However, how populations of neurons in the RNN collectively achieve such performance still remains poorly understood. To tackle this problem, we investigated learned representations in RNNs by extracting low-dimensional, neural trajectories that summarize the activity of a population of neurons in the network during individual syntheses of handwritings. The neural trajectories show that different writing styles are encoded in different subspaces inside an internal space of the network. Within each subspace, different characters of the same style are represented as different state dynamics. These results demonstrate the effectiveness of analyzing the neural trajectory for intuitive understanding of how the RNNs work.

Motivation & Objective

• To understand how RNNs collectively represent handwriting styles and character dynamics at the population level.
• To investigate the internal neural representations underlying style transfer in RNN-based handwriting synthesis.
• To determine whether different writing styles are encoded in distinct subspaces of the RNN's hidden space.
• To examine how individual characters are represented as dynamic state trajectories within a given style subspace.
• To identify the functional specialization of individual recurrent layers in style and character representation.

Proposed method

• Applied Gaussian-Process Factor Analysis (GPFA) to reduce dimensionality of LSTM activation time-series from three recurrent layers.
• Preprocessed LSTM activations using a median filter (kernel size 3) to resolve rank deficiency in covariance matrices.
• Fitted GPFA models to 120 trials across multiple priming conditions and target texts using the EM algorithm.
• Extracted low-dimensional neural trajectories summarizing population activity during handwriting synthesis.
• Used KL divergence to quantitatively compare neural trajectory distributions across different priming and text conditions.
• Visualized and compared trajectories across layers to assess style and character representation.

Experimental results

Research questions

• RQ1 Do different handwriting styles correspond to distinct subspaces in the RNN's internal neural space?
• RQ2 How are individual characters represented within a given writing style's subspace?
• RQ3 Which recurrent layer plays the most critical role in maintaining writing style during synthesis?
• RQ4 Are neural trajectories for the same character consistent across different texts and writers?
• RQ5 How do priming conditions influence the structure of neural trajectories in the RNN's hidden space?

Key findings

• Different handwriting styles are encoded in distinct subspaces within the RNN's internal neural space, as evidenced by significant differences in neural trajectory distributions.
• Neural trajectories for the same character vary across writers but remain consistent within a given style, indicating dynamic encoding of character-specific patterns.
• The second recurrent layer shows the most pronounced differences in neural trajectories between writing styles, suggesting a key role in style maintenance.
• Statistical analysis (Mann-Whitney U-test, p < 0.001) confirms that trajectory differences between styles are significantly greater than those within the same style.
• The first and third recurrent layers also exhibit significant differences in trajectory patterns across styles, indicating distributed representation of style information.
• Characters of the same style are represented as distinct state dynamics within their respective subspace, demonstrating that character-level information is embedded in temporal dynamics.