[Paper Review] Predictive Chemistry Augmented with Text Retrieval

Yujie Qian, Zhening Li+3 more|arXiv (Cornell University)|Dec 8, 2023

Advanced Text Analysis Techniques4 citations

TL;DR

TextReact proposes a novel framework that enhances predictive chemistry models by retrieving and aligning relevant scientific literature texts with molecular reaction representations. By integrating text retrieval with a masked language modeling objective, TextReact achieves state-of-the-art performance, improving top-1 accuracy by 58.4% on reaction condition recommendation and 13.6–15.7% on one-step retrosynthesis compared to models trained on molecular data alone.

ABSTRACT

This paper focuses on using natural language descriptions to enhance predictive models in the chemistry field. Conventionally, chemoinformatics models are trained with extensive structured data manually extracted from the literature. In this paper, we introduce TextReact, a novel method that directly augments predictive chemistry with texts retrieved from the literature. TextReact retrieves text descriptions relevant for a given chemical reaction, and then aligns them with the molecular representation of the reaction. This alignment is enhanced via an auxiliary masked LM objective incorporated in the predictor training. We empirically validate the framework on two chemistry tasks: reaction condition recommendation and one-step retrosynthesis. By leveraging text retrieval, TextReact significantly outperforms state-of-the-art chemoinformatics models trained solely on molecular data.

Motivation & Objective

• To improve predictive chemistry models by incorporating unstructured scientific text from the literature, rather than relying solely on structured molecular data.
• To address the limitation of existing chemoinformatics models that operate only on molecular encodings, which may miss contextual cues from published reactions.
• To develop a retrieval-augmented framework that fuses molecular representations with relevant textual descriptions to enhance prediction accuracy and generalization.
• To validate the approach on two core chemistry tasks: reaction condition recommendation and one-step retrosynthesis.
• To demonstrate that text retrieval can significantly improve model performance even when gold-standard texts are excluded from training, indicating robust generalization.

Proposed method

• TextReact employs a SMILES-to-text retriever that maps a given reaction (in SMILES format) to relevant text descriptions from an unlabeled corpus using contrastive learning.
• The framework uses a text-augmented predictor that jointly encodes the input reaction and retrieved text embeddings to generate predictions.
• An auxiliary masked language modeling (MLM) objective is incorporated during training to align the representations of the reaction and its corresponding text, improving cross-modal understanding.
• The model is trained with a data augmentation strategy that removes the closest textual descriptions during training to simulate unseen reaction classes and improve generalization.
• The retrieved texts are concatenated and jointly encoded rather than processed independently, enabling the model to reason over multiple evidence sources simultaneously.
• Pretrained SciBERT is used to initialize the text encoder, enhancing performance through prior exposure to scientific language.

Figure 1: Predictive chemistry augmented with text retrieval. For example, given the task of reaction condition recommendation, we retrieve texts relevant to the input reaction to provide additional evidence for the model’s prediction. The two retrieved texts describe similar reactions to the input

Experimental results

Research questions

• RQ1 Can retrieving relevant scientific text descriptions from the literature significantly improve the performance of predictive chemistry models?
• RQ2 How does the integration of retrieved text with molecular representations affect model generalization, especially on unseen reaction types?
• RQ3 Does the auxiliary masked language modeling objective improve the alignment between molecular and textual representations in the context of chemistry prediction?
• RQ4 How does the performance of TextReact compare to state-of-the-art models trained exclusively on structured molecular data?
• RQ5 To what extent does the quality of retrieved text (e.g., similarity to input reaction) influence prediction accuracy?

Key findings

• TextReact achieves a 58.4% absolute improvement in top-1 accuracy for reaction condition recommendation compared to state-of-the-art models trained only on molecular data.
• For one-step retrosynthesis, TextReact improves performance by 13.6–15.7% in top-1 accuracy over baseline models trained on molecular data alone.
• The model maintains strong performance even when retrieving from a corpus that excludes the gold-standard text, demonstrating robustness and generalization to unseen reaction classes.
• The ablation study shows that removing the masked language modeling (MLM) objective reduces performance, confirming its effectiveness in aligning text and molecular representations.
• Jointly encoding retrieved texts (rather than ensembling separate predictions) leads to better performance, indicating the benefit of end-to-end integration.
• TextReact achieves superior results with only 10% of the training data compared to a standard Transformer baseline, highlighting data efficiency.

Figure 2: One-step retrosynthesis, another predictive chemistry task studied in this paper.