[Paper Review] RxnScribe: A Sequence Generation Model for Reaction Diagram Parsing

Yujie Qian, Jiang Guo+3 more|arXiv (Cornell University)|May 19, 2023

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

RxnScribe introduces a sequence generation model for end-to-end parsing of chemical reaction diagrams into structured reaction schemes, formulating the task as sequence-to-sequence prediction using a tokenized representation of entities and roles. It achieves an 80.0% soft match F1 score on a diverse dataset of 1,378 diagrams, significantly outperforming prior methods, especially on complex graph-style diagrams, with strong generalization enabled by compositional data augmentation.

ABSTRACT

Reaction diagram parsing is the task of extracting reaction schemes from a diagram in the chemistry literature. The reaction diagrams can be arbitrarily complex, thus robustly parsing them into structured data is an open challenge. In this paper, we present RxnScribe, a machine learning model for parsing reaction diagrams of varying styles. We formulate this structured prediction task with a sequence generation approach, which condenses the traditional pipeline into an end-to-end model. We train RxnScribe on a dataset of 1,378 diagrams and evaluate it with cross validation, achieving an 80.0% soft match F1 score, with significant improvements over previous models. Our code and data are publicly available at https://github.com/thomas0809/RxnScribe.

Motivation & Objective

• To develop a general-purpose machine learning model that robustly parses diverse chemical reaction diagrams from the literature.
• To address the challenge of error propagation in traditional pipelined approaches by unifying entity recognition and relationship prediction into an end-to-end sequence generation framework.
• To construct a large-scale, expert-annotated dataset of 1,378 reaction diagrams spanning four styles (single-line, multiple-line, tree, graph) for training and evaluation.
• To improve model generalization and performance on rare or complex diagram types using a novel compositional data augmentation strategy.

Proposed method

• Formulate reaction diagram parsing as a sequence generation task, where each reaction is represented as a sequence of tokens encoding entity type, role (reactant/product/condition), bounding box coordinates, and content.
• Define a discrete sequence representation that encodes molecular structures, text, and reaction roles in a structured, sequential format for model input.
• Train a transformer-based sequence generation model (e.g., T5 or similar) to predict the full reaction sequence conditioned on the diagram image.
• Use off-the-shelf tools—MolScribe for molecular structure recognition and OCR for text extraction—on the predicted bounding boxes to generate final SMILES and text outputs.
• Apply a compositional data augmentation strategy that composes simple diagrams into more complex ones to increase training data diversity and improve performance on rare styles.
• Leverage five-fold cross-validation for evaluation, focusing on reaction structure prediction accuracy using soft match F1 score.

Figure 1 : Examples of reaction diagrams in chemistry literature. We summarize four common styles of reaction diagrams: single-line, multiple-line, tree, and graph. The example diagrams are adapted with permission from Faizi et al. 1 [Copyright © 2016 American Chemical Society], Armitage et al. 2 [C

Experimental results

Research questions

• RQ1 Can a sequence generation model effectively unify the parsing of diverse reaction diagram styles into a single end-to-end framework?
• RQ2 How does the proposed sequence representation compare to traditional pipelined approaches in terms of robustness and error propagation?
• RQ3 To what extent does compositional data augmentation improve model generalization, especially on low-resource diagram styles like graph or tree structures?
• RQ4 What is the performance ceiling of sequence-based parsing on real-world, complex reaction diagrams from the literature?
• RQ5 How does model performance scale with increasing training data size and diagram complexity?

Key findings

• RxnScribe achieves an 80.0% soft match F1 score on the full dataset, representing a significant improvement over previous models, which scored below 10% on the same benchmark.
• The model performs best on single-line diagrams (91.0% F1), with strong performance on graph-style diagrams (65.9% F1), demonstrating robustness across styles despite their complexity.
• Compositional data augmentation significantly improves performance on complex styles (tree, multiple-line, graph), but provides less benefit on simpler single-line diagrams.
• Performance improves consistently with more training data, especially on diagrams with multiple reactions, indicating strong potential for further gains with larger datasets.
• The model generalizes well even on low-resource styles—achieving over 60% F1 on the smallest group (graph-style, 102 examples)—highlighting the effectiveness of the sequence generation approach.

Figure 2 : Overview of reaction diagram parsing. The input is a reaction diagram, and the output is a list of reactions. The example diagram is from a journal article. 1