[Paper Review] Deep Confident Steps to New Pockets: Strategies for Docking Generalization

Gabriele Corso, Arthur Deng+4 more|PubMed|Feb 28, 2024

Protein Structure and Dynamics30 references19 citations

TL;DR

The paper introduces DockGen, a new benchmark for blind docking generalization, shows ML docking models generalize poorly to unseen pockets, and presents scaling and Confidence Bootstrapping to improve generalization and achieve state-of-the-art results.

ABSTRACT

Accurate blind docking has the potential to lead to new biological breakthroughs, but for this promise to be realized, docking methods must generalize well across the proteome. Existing benchmarks, however, fail to rigorously assess generalizability. Therefore, we develop DockGen, a new benchmark based on the ligand-binding domains of proteins, and we show that existing machine learning-based docking models have very weak generalization abilities. We carefully analyze the scaling laws of ML-based docking and show that, by scaling data and model size, as well as integrating synthetic data strategies, we are able to significantly increase the generalization capacity and set new state-of-the-art performance across benchmarks. Further, we propose Confidence Bootstrapping, a new training paradigm that solely relies on the interaction between diffusion and confidence models and exploits the multi-resolution generation process of diffusion models. We demonstrate that Confidence Bootstrapping significantly improves the ability of ML-based docking methods to dock to unseen protein classes, edging closer to accurate and generalizable blind docking methods.

Motivation & Objective

• Motivate the need for evaluating generalization in blind docking across diverse protein pockets.
• Propose DockGen as a domain-level benchmark to assess cross-domain generalization beyond training pockets.
• Analyze scaling laws of ML-based docking to understand data/model growth effects on generalization.
• Introduce synthetic data augmentation to expand pocket diversity and study its impact.
• Present Confidence Bootstrapping to improve generalization by feedback between diffusion and confidence models.

Proposed method

• Develop DockGen benchmark by clustering protein domains (ECOD) and creating validation/test sets across 141/189 complexes from Binding MOAD with filtering.
• Evaluate baseline ML and search-based docking methods on DockGen to quantify generalization gaps.
• Scale data and model size to study generalization trends; introduce van der Mer-inspired synthetic sidechain ligands to augment training.
• Propose DiffDock-L by combining larger models and augmented data to achieve new SOTA on DockGen (and other benchmarks).
• Introduce Confidence Bootstrapping: a self-training scheme where a diffusion model generates poses, a confidence model scores them, and feedback updates early diffusion steps.
• Formulate iterative optimization where diffusion scores are updated via rollout and confidence-guided reweighting across iterations.

Figure 1: Visual representation of the Confidence Bootstrapping training scheme. The dashed lines represent the reverse diffusion generation rollouts that the model executes. The dotted lines illustrate the bootstrapping feedback from the confidence model that is used to update the likelihood of the

Experimental results

Research questions

• RQ1 How well do existing ML-based docking methods generalize to unseen protein pocket domains?
• RQ2 What is the impact of increasing training data and model size on generalization in blind docking?
• RQ3 Can synthetic data augmentation broaden pocket diversity and improve docking generalization?
• RQ4 Can a confidence-guided bootstrapping scheme improve diffusion-based docking on unseen protein classes?
• RQ5 What is the resulting performance of a larger, augmented diffusion model on the DockGen benchmark?

Key findings

• DockGen reveals strong generalization gaps for existing ML docking methods on unseen pockets.
• DiffDock-L improves top-1 success rate from 7.1% to 22.6% on DockGen, surpassing a strong baseline and achieving state-of-the-art results.
• Augmenting training data with MOAD and employing van der Mer-inspired synthetic ligands increases pocket diversity and modestly boosts performance.
• Larger model sizes (e.g., 30M parameters) yield generalization gains when paired with data augmentation.
• Confidence Bootstrapping significantly boosts performance of DiffDock-S on DockGen-clusters from 9.8% to 24.0%, with >30% in half of the clusters.
• Across DockGen, confidence-boosted diffusion approaches outperform traditional search methods under high exhaustiveness.

Figure 2: A. An example of the superimposition of the pockets of two proteins in PDBBind, 1QXZ in pink and 5M4Q in cyan, that share a very similar binding pocket structure (a bound ligand is shown in red), but have only 22% sequence similarity. While sequence similarity splits would classify them in