[Paper Review] CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph

Haitao Lin, Guojiang Zhao+8 more|arXiv (Cornell University)|Jun 16, 2024

Bioinformatics and Genomic NetworksBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

CBGBench proposes a unified benchmark for structure-based drug design (SBDD) by framing molecular generation as 3D heterogeneous graph completion—akin to fill-in-the-blank of a protein-ligand binding graph. It standardizes evaluation across generation, interaction, geometry, and substructure metrics, enabling fair comparison of state-of-the-art models and revealing that CNN-based and autoregressive methods remain competitive, while domain knowledge integration shows limited gains.

ABSTRACT

Structure-based drug design (SBDD) aims to generate potential drugs that can bind to a target protein and is greatly expedited by the aid of AI techniques in generative models. However, a lack of systematic understanding persists due to the diverse settings, complex implementation, difficult reproducibility, and task singularity. Firstly, the absence of standardization can lead to unfair comparisons and inconclusive insights. To address this dilemma, we propose CBGBench, a comprehensive benchmark for SBDD, that unifies the task as a generative heterogeneous graph completion, analogous to fill-in-the-blank of the 3D complex binding graph. By categorizing existing methods based on their attributes, CBGBench facilitates a modular and extensible framework that implements various cutting-edge methods. Secondly, a single task on extit{de novo} molecule generation can hardly reflect their capabilities. To broaden the scope, we have adapted these models to a range of tasks essential in drug design, which are considered sub-tasks within the graph fill-in-the-blank tasks. These tasks include the generative designation of extit{de novo} molecules, linkers, fragments, scaffolds, and sidechains, all conditioned on the structures of protein pockets. Our evaluations are conducted with fairness, encompassing comprehensive perspectives on interaction, chemical properties, geometry authenticity, and substructure validity. We further provide the pre-trained versions of the state-of-the-art models and deep insights with analysis from empirical studies. The codebase for CBGBench is publicly accessible at \url{https://github.com/Edapinenut/CBGBench}.

Motivation & Objective

• To address the lack of standardized, comprehensive evaluation in structure-based drug design (SBDD) due to inconsistent protocols, implementation complexity, and task singularity.
• To unify diverse SBDD methods into a single modular framework by reformulating molecular generation as heterogeneous graph completion (i.e., fill-in-the-blank of the 3D binding graph).
• To extend evaluation beyond de novo molecule generation to include sub-tasks critical in lead optimization: linker, fragment, scaffold, and sidechain design.
• To enable fair, comprehensive, and reproducible benchmarking using unified metrics across chemical properties, interaction patterns, geometric validity, and substructure fidelity.
• To validate the generalizability of the benchmark by applying pretrained models to real-world targets (e.g., ADRB1, DRD3) and assess performance on experimental actives.

Proposed method

• Reformulate SBDD as a 3D heterogeneous graph completion task, where the protein-ligand complex is represented as a graph with nodes for atoms and edges for interactions, and the molecule is generated by 'filling in the blank' of missing atom and bond information.
• Categorize existing methods along three dichotomies: (i) voxelized vs. continuous position generation, (ii) one-shot vs. autoregressive generation, and (iii) domain-knowledge-based vs. full-data-driven learning.
• Implement a modular, extensible codebase integrating state-of-the-art models (e.g., Pocket2Mol, TargetDiff, D3FG, DiffBP) under a unified framework for fair comparison.
• Introduce a comprehensive evaluation protocol including: (i) chemical properties (QED, SA, LogP, LPSK), (ii) interaction types and Vina docking energy, (iii) geometric metrics (bond lengths, angles, clashes), and (iv) substructure validity (pharmacophores, rings, functional groups).
• Extend the framework to five tasks: de novo molecule generation, linker design, fragment design, sidechain optimization, and scaffold hopping, all conditioned on protein pocket structures.
• Validate the benchmark on real-world targets (ADRB1, DRD3) using ECFP fingerprints (t-SNE visualization) and binding affinity distributions (Vina score, LBE), comparing generated molecules to experimental actives and random controls.

Experimental results

Research questions

• RQ1 Can a unified graph completion framework fairly compare diverse SBDD methods across different architectures and training paradigms?
• RQ2 How do state-of-the-art generative models perform across multiple evaluation dimensions (chemical, geometric, interaction, substructure) in a standardized protocol?
• RQ3 To what extent can existing SBDD models be generalized to lead optimization tasks such as linker or scaffold design?
• RQ4 How well do the benchmark’s evaluation metrics correlate with real-world binding affinities on experimental targets?
• RQ5 What are the relative strengths and limitations of data-driven vs. knowledge-augmented models in protein-ligand binding prediction?

Key findings

• CNN-based methods using density maps remain highly competitive in target-aware molecular generation, outperforming some diffusion-based alternatives in chemical space consistency.
• Autoregressive models require explicit modeling of chemical bond patterns to achieve high-quality generation, as failure to do so leads to structural artifacts.
• Despite integration of physical and chemical domain knowledge, recent models show only marginal improvements, indicating that effective knowledge distillation remains a key challenge.
• Most evaluated models generalize well to lead optimization tasks, with linker design being the most feasible and scaffold hopping the most difficult.
• The benchmark’s evaluation protocol shows strong consistency and generalizability, as evidenced by similar performance rankings on both in-house and real-world targets (e.g., ADRB1, DRD3).
• D3FG and TargetDiff demonstrate superior and stable performance in generating molecules with favorable binding affinities (low Vina score, low LBE) on real targets, while DiffBP, FLAG, and Pocket2Mol show potential for lead discovery due to small molecular weight and high stability.