[Paper Review] Benchmarking Generated Poses: How Rational is Structure-based Drug Design with Generative Models?

Charles B. Harris, Kieran Didi+5 more|arXiv (Cornell University)|Aug 14, 2023

Protein Structure and Dynamics20 citations

TL;DR

The paper introduces PoseCheck, a biophysical benchmark suite for evaluating generated binding poses in structure-based drug design, showing that generated poses often have physical violations and that redocking can mask these issues.

ABSTRACT

Deep generative models for structure-based drug design (SBDD), where molecule generation is conditioned on a 3D protein pocket, have received considerable interest in recent years. These methods offer the promise of higher-quality molecule generation by explicitly modelling the 3D interaction between a potential drug and a protein receptor. However, previous work has primarily focused on the quality of the generated molecules themselves, with limited evaluation of the 3D molecule \emph{poses} that these methods produce, with most work simply discarding the generated pose and only reporting a "corrected" pose after redocking with traditional methods. Little is known about whether generated molecules satisfy known physical constraints for binding and the extent to which redocking alters the generated interactions. We introduce PoseCheck, an extensive analysis of multiple state-of-the-art methods and find that generated molecules have significantly more physical violations and fewer key interactions compared to baselines, calling into question the implicit assumption that providing rich 3D structure information improves molecule complementarity. We make recommendations for future research tackling identified failure modes and hope our benchmark can serve as a springboard for future SBDD generative modelling work to have a real-world impact.

Motivation & Objective

• Assess the quality of generated binding poses beyond 2D molecule metrics in SBDD.
• Evaluate whether 3D generative models meet known biophysical constraints during binding.
• Characterize the impact of redocking on pose quality and interaction accuracy.
• Identify failure modes and provide recommendations to improve SBDD generative methods.

Proposed method

• Introduce PoseCheck comprising pipeline-wide and component-wise metrics for generated and redocked poses.
• Use interaction fingerprinting (ProLIF) to compare hydrogen bonds, vdW contacts, and hydrophobic interactions.
• Quantify steric clashes using a 0.5 Å tolerance based on van der Waals radii.
• Compute strain energy via the difference between relaxed and generated poses using UFF in RDKit.
• Evaluate RMSD between generated and redocked poses with QuickVina2 across five methods on CrossDocked2020 test set.
• Analyze hydrogen bonding and other interactions to compare generated poses to CrossDocked references.

Figure 1: Top: Overview of a conventional pipeline of SBDD with 3D generative modelling. A generative model is usually trained using experimental or synthetic protein-ligand complexes, from which new molecules and poses can be sampled de novo . Typically, generated poses are discarded and redocked i

Experimental results

Research questions

• RQ1 Do generated poses satisfy biophysical constraints (hydrogen bonding, steric compatibility) as often as the CrossDocked references?
• RQ2 How does redocking affect the observed pose quality and interaction patterns of generated molecules?
• RQ3 Which SBDD generative methods produce physically plausible poses with fewer clashes and lower strain energy?
• RQ4 What are the primary failure modes in current 3D generative SBDD models?

Key findings

• All methods show high RMSD between generated and redocked poses (median not below 2 Å).
• Generated poses exhibit significantly fewer hydrogen bond donors/acceptors than the baseline, indicating weaker hydrogen bonding networks.
• Steric clashes are more prevalent in diffusion-based methods, with redocking reducing but not eliminating severe clashes.
• Strain energy of generated poses is generally much higher than the CrossDocked baseline, especially for diffusion models and LiGAN.
• Redocking can mask fundamental pose implausibilities rather than correcting them in many cases.
• LiGAN shows relatively better interaction similarity before redocking but degenerates after redocking.

Figure 2: RSMD between the generated and redocked poses using the popular docking framework Vina. Left: Violin plots of all redocking RMSDs. CrossDocked violin is where we have redocking the poses from the original dataset. Right: Illustrative examples (generated in magenta, redocked in green). We o