[Paper Review] Teaching Machines to Code: Neural Markup Generation with Visual Attention

Sumeet S. Singh|arXiv (Cornell University)|Feb 15, 2018

Multimodal Machine Learning Applications33 references4 citations

TL;DR

This paper proposes a neural transducer model with visual attention that generates syntactically correct LaTeX markup from math formula images, achieving a state-of-the-art BLEU score of 89% on the Im2Latex benchmark. The model uses a CNN encoder with spatial pooling and an attention-augmented LSTM decoder to map image features to LaTeX sequences, while attention visualization enables accurate symbol localization without bounding box supervision.

ABSTRACT

We present a neural transducer model with visual attention that learns to generate LaTeX markup of a real-world math formula given its image. Applying sequence modeling and transduction techniques that have been very successful across modalities such as natural language, image, handwriting, speech and audio; we construct an image-to-markup model that learns to produce syntactically and semantically correct LaTeX markup code over 150 words long and achieves a BLEU score of 89%; improving upon the previous state-of-art for the Im2Latex problem. We also demonstrate with heat-map visualization how attention helps in interpreting the model and can pinpoint (detect and localize) symbols on the image accurately despite having been trained without any bounding box data.

Motivation & Objective

• To develop an end-to-end image-to-LaTeX markup model that generates semantically and syntactically correct LaTeX code from math formula images.
• To improve upon the previous state-of-the-art BLEU score for the Im2Latex problem.
• To demonstrate that visual attention in the model enables interpretable localization of symbols in images without requiring bounding box annotations.
• To analyze architectural variations and loss functions in the context of sequence-to-sequence markup generation.
• To explore the potential of attention mechanisms for zero-shot object detection in formula images.

Proposed method

• Employs an encoder-decoder architecture with visual attention, where a deep CNN encodes input images into a visual feature grid.
• Applies spatial pooling to reduce feature map resolution, creating localized regional encodings with narrow receptive fields.
• Uses an LSTM-based decoder with a deep output layer and attention mechanism to generate LaTeX tokens step-by-step.
• Applies beam search decoding with a custom loss function to optimize sequence generation.
• Visualizes attention weights as heatmaps to interpret model behavior and localize symbols in the image.
• Trains the model end-to-end using supervised learning on a large-scale dataset of rendered LaTeX formulas.

Experimental results

Research questions

• RQ1 Can a neural sequence-to-sequence model with visual attention generate high-accuracy LaTeX markup from math formula images without explicit bounding box supervision?
• RQ2 How does architectural design—particularly pooling and attention—impact the performance and interpretability of image-to-markup models?
• RQ3 To what extent can attention mechanisms localize individual symbols in a formula image, even when trained without any object detection annotations?
• RQ4 What is the impact of different loss functions and model variants on the final BLEU score and generalization performance?
• RQ5 Can attention-based attention visualization serve as a proxy for object detection in formula images, enabling zero-shot localization?

Key findings

• The model achieves a BLEU score of 89.0% on the I2L-140K dataset, representing a 1.27% improvement over the previous state-of-the-art.
• Attention visualization successfully localizes individual symbols in formula images with high precision, even though the model was trained without any bounding box data.
• The model correctly renders 99.94% of predicted LaTeX sequences, indicating strong syntactic and semantic correctness.
• The use of spatial pooling with narrow receptive fields enables the model to focus on localized image regions, which supports accurate attention-based localization.
• The model generalizes well to unseen formulas, with 70.37% of predictions matching the ground truth in visual similarity (image match) evaluation.
• In some cases, the model's output is more accurate than the original formula, suggesting potential for automated formula correction.