[Paper Review] Grounding Large Language Models in Reaction Knowledge Graphs for Synthesis Retrieval

Olga Bunkova, Lorenzo Di Fruscia+4 more|arXiv (Cornell University)|Jan 22, 2026

Machine Learning in Materials Science0 citations

TL;DR

The paper investigates grounding LLMs with a reaction knowledge graph by formulating reaction-path retrieval as Text2Cypher generation, comparing prompting strategies and a checklist-based correction loop, and providing a reproducible evaluation setup.

ABSTRACT

Large Language Models (LLMs) can aid synthesis planning in chemistry, but standard prompting methods often yield hallucinated or outdated suggestions. We study LLM interactions with a reaction knowledge graph by casting reaction path retrieval as a Text2Cypher (natural language to graph query) generation problem, and define single- and multi-step retrieval tasks. We compare zero-shot prompting to one-shot variants using static, random, and embedding-based exemplar selection, and assess a checklist-driven validator/corrector loop. To evaluate our framework, we consider query validity and retrieval accuracy. We find that one-shot prompting with aligned exemplars consistently performs best. Our checklist-style self-correction loop mainly improves executability in zero-shot settings and offers limited additional retrieval gains once a good exemplar is present. We provide a reproducible Text2Cypher evaluation setup to facilitate further work on KG-grounded LLMs for synthesis planning. Code is available at https://github.com/Intelligent-molecular-systems/KG-LLM-Synthesis-Retrieval.

Motivation & Objective

• Motivate grounding LLMs in structured chemical data to reduce hallucinations and outdated knowledge during synthesis planning.
• Develop a Text2Cypher pipeline to translate natural language questions into executable graph queries over a reaction knowledge graph.
• Systematically compare zero-shot and one-shot prompting strategies, including static, random, and embedding-based exemplar selection.
• Introduce a lightweight checklist-based validator/corrector loop to improve executability of generated queries, especially in zero-shot settings.
• Provide a reproducible evaluation setup (datasets, prompts, metrics) to benchmark KG-grounded LLMs for synthesis retrieval.

Proposed method

• Construct a bipartite reaction knowledge graph (KG) from USPTO reactions in SMILES, storing reactions and molecules as distinct node types.
• Cast retrosynthesis retrieval as a Text2Cypher task, generating Cypher queries from natural language queries and validating executability in Neo4j.
• Design five prompt variants per setting (single- and multi-step) and compare zero-shot to one-shot prompting with static, dynamic random, and embedding-based exemplar selection.
• Implement a Chain-of-Verification (CoVe) style loop where a validator checks executability, and a corrector refines invalid queries up to three attempts.
• Evaluate using query-text similarity (BLEU, METEOR, ROUGE-L) and retrieval metrics (precision, recall, F1; exact-path and partial-path recalls for multi-step routes).
• Use GPT-4.1-mini (deterministic) for generation and provide an open-source Text2Cypher evaluation setup and code.

Experimental results

Research questions

• RQ1 Can LLMs generate executable Cypher queries that yield correct single-step retrosynthesis retrieval over a reaction KG?
• RQ2 Can LLMs produce correct multi-step retrosynthesis routes (paths up to length 4) when queried via natural language?
• RQ3 How do zero-shot vs one-shot prompting and exemplar selection strategies affect retrieval accuracy and executability?
• RQ4 Does a checklist-based validator/corrector loop improve query executability and retrieval quality, and under what conditions is it most beneficial?

Key findings

• One-shot prompting with aligned exemplars consistently yields the best retrieval performance.
• The CoVe-style self-correction loop mainly improves executability in zero-shot settings and offers limited gains when a good exemplar is provided.
• Text-to-text similarity metrics (BLEU, METEOR, ROUGE-L) are poor proxies for retrieval accuracy in this task.
• Moving from zero-shot to one-shot prompting reduces common retrieval errors, especially endpoint anchoring and traversal-direction violations in multi-step tasks.
• A task-specific validator is a bottleneck; generic checklists miss many task-specific failures, suggesting the need for schema-aware validators.
• The framework provides a reproducible Text2Cypher evaluation setup and baseline results to guide KG-grounded LLMs in synthesis planning.