[Paper Review] Benchmarking structure-based three-dimensional molecular generative models using GenBench3D: ligand conformation quality matters

Benoît Baillif, Jason C. Cole+2 more|arXiv (Cornell University)|Jul 5, 2024

Genetics, Bioinformatics, and Biomedical Research4 citations

TL;DR

This paper introduces GenBench3D, a benchmark for structure-based 3D molecular generative models that evaluates ligand conformation quality using the novel Validity3D metric, which assesses bond lengths and valence angles against Cambridge Structural Database references. Only 0–11% of generated molecules had valid conformations, but local relaxation improved Validity3D by at least 40%, revealing that raw-generated molecules often overestimate binding affinity, especially under Vina scoring.

ABSTRACT

Three-dimensional (3D) deep molecular generative models offer the advantage of goal-directed generation based on 3D-dependent properties, such as binding affinity for structure-based design within binding pockets. Traditional benchmarks created to evaluate SMILES or molecular graphs generators, such as GuacaMol or MOSES, are limited to evaluate 3D generators as they do not assess the quality of the generated molecular conformation. In this work, we hence developed GenBench3D, which implements a new benchmark for models producing molecules within a binding pocket. Our main contribution is the Validity3D metric, evaluating the conformation quality using the likelihood of bond lengths and valence angles based on reference values observed in the Cambridge Structural Database. The LiGAN, 3D-SBDD, Pocket2Mol, TargetDiff, DiffSBDD and ResGen models were benchmarked. We show that only between 0% and 11% of generated molecules have valid conformations. Performing local relaxation of generated molecules in the pocket considerably improved the Validity3D for all models by a minimum increase of 40%. For LiGAN, 3D-SBDD, or TargetDiff, the set of valid relaxed molecules shows on average higher Vina score (i.e. worse) than the set of raw generated molecules, indicating that the binding affinity of raw generated molecules might be overestimated. Using the other scoring functions, that give higher importance to ligand strain, only yield improved scores when using valid relaxed molecules. Using valid relaxed molecules, TargetDiff and Pocket2Mol show better median Vina, Glide and Gold PLP scores than other models. We have publicly released GenBench3D on GitHub for broader use: https://github.com/bbaillif/genbench3d

Motivation & Objective

• To address the lack of benchmarks that evaluate 3D molecular conformation quality in structure-based generative models.
• To identify and quantify the prevalence of geometrically invalid conformations in generated molecules within binding pockets.
• To assess whether conformational relaxation improves both geometric validity and binding affinity predictions.
• To compare the performance of six 3D generative models—LiGAN, 3D-SBDD, Pocket2Mol, TargetDiff, DiffSBDD, and ResGen—using a standardized benchmark.
• To release GenBench3D publicly to enable reproducible evaluation of future 3D molecular generation models.

Proposed method

• Developed GenBench3D, a benchmark framework for evaluating 3D molecular generative models in binding pocket contexts.
• Introduced the Validity3D metric, which computes the likelihood of bond lengths and valence angles relative to reference values from the Cambridge Structural Database.
• Generated molecules using six state-of-the-art 3D generative models within predefined binding pockets.
• Performed local conformational relaxation using molecular mechanics (e.g., OPLS4 force field) to improve geometric quality.
• Evaluated binding affinity using multiple scoring functions: AutoDock Vina, Glide, and Gold PLP.
• Compared raw generated molecules with their relaxed counterparts across geometric validity and scoring function performance.

Experimental results

Research questions

• RQ1 How many of the generated molecules by 3D generative models have geometrically valid conformations when placed in a binding pocket?
• RQ2 To what extent does local conformational relaxation improve the geometric validity of generated molecules?
• RQ3 Does the binding affinity of raw-generated molecules tend to be overestimated due to poor conformational quality?
• RQ4 Which 3D generative model produces the most geometrically valid and highly scoring molecules after relaxation?
• RQ5 How do different scoring functions (Vina, Glide, Gold PLP) respond to conformational relaxation in terms of binding affinity prediction accuracy?

Key findings

• Only 0% to 11% of molecules generated by the six models had valid conformations before relaxation, as measured by the Validity3D metric.
• Local conformational relaxation improved Validity3D by a minimum of 40% across all models, with some models showing significantly higher gains.
• For LiGAN, 3D-SBDD, and TargetDiff, the Vina scores of relaxed molecules were worse on average than those of raw generated molecules, indicating that raw scores may overestimate binding affinity.
• When using valid, relaxed molecules, TargetDiff and Pocket2Mol achieved better median scores across Vina, Glide, and Gold PLP scoring functions than other models.
• Scoring functions that penalize ligand strain (e.g., Glide, Gold PLP) showed improved performance only when applied to relaxed, valid conformations.
• GenBench3D has been released on GitHub to support reproducible benchmarking of future 3D molecular generative models.