[Paper Review] HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddle

Guoxia Wang, Xiaomin Fang+7 more|arXiv (Cornell University)|Jul 12, 2022

Machine Learning in Bioinformatics30 citations

TL;DR

HelixFold implements AlphaFold2 end-to-end in PaddlePaddle, achieving faster training with Branch Parallelism and memory optimizations while maintaining comparable accuracy to AlphaFold2 on CASP14 and CAMEO.

ABSTRACT

Accurate protein structure prediction can significantly accelerate the development of life science. The accuracy of AlphaFold2, a frontier end-to-end structure prediction system, is already close to that of the experimental determination techniques. Due to the complex model architecture and large memory consumption, it requires lots of computational resources and time to implement the training and inference of AlphaFold2 from scratch. The cost of running the original AlphaFold2 is expensive for most individuals and institutions. Therefore, reducing this cost could accelerate the development of life science. We implement AlphaFold2 using PaddlePaddle, namely HelixFold, to improve training and inference speed and reduce memory consumption. The performance is improved by operator fusion, tensor fusion, and hybrid parallelism computation, while the memory is optimized through Recompute, BFloat16, and memory read/write in-place. Compared with the original AlphaFold2 (implemented with Jax) and OpenFold (implemented with PyTorch), HelixFold needs only 7.5 days to complete the full end-to-end training and only 5.3 days when using hybrid parallelism, while both AlphaFold2 and OpenFold take about 11 days. HelixFold saves 1x training time. We verified that HelixFold's accuracy could be on par with AlphaFold2 on the CASP14 and CAMEO datasets. HelixFold's code is available on GitHub for free download: https://github.com/PaddlePaddle/PaddleHelix/tree/dev/apps/protein_folding/helixfold, and we also provide stable web services on https://paddlehelix.baidu.com/app/drug/protein/forecast.

Motivation & Objective

• Aim to reduce training time and memory usage of AlphaFold2 without sacrificing accuracy
• Develop a complete training and inference pipeline for AlphaFold2 in PaddlePaddle (HelixFold)
• Introduce novel parallelism and memory-optimization techniques to enable efficient end-to-end training

Proposed method

• Fuse multiple operators and tensors to reduce CPU scheduling overhead (Tensor Fusion and Fused Gated Self-Attention)
• Propose Branch Parallelism (BP) to parallelize Evoformer’s MSA and Pair stacks across devices
• Adopt hybrid parallelism (BP-DAP-DP) to improve throughput with data parallelism and memory efficiency
• Apply memory optimizations: Recomputation, BFloat16 activations, in-place memory read/write, and Subbatch/DAP techniques
• Fuse parameters, gradients, and optimizer states to reduce kernel launches and memory fragmentation
• Utilize Subbatch and DAP to manage peak memory for very long sequences

Experimental results

Research questions

• RQ1 Can HelixFold match AlphaFold2's accuracy on CASP14 and CAMEO while reducing training time?
• RQ2 How do fusion, branch parallelism, and memory optimization techniques affect training throughput and memory footprint?
• RQ3 What are the comparative benefits of BP-DAP-DP hybrid parallelism vs existing parallelism strategies for Evoformer?
• RQ4 What is the end-to-end training time and resource cost reduction when using PaddlePaddle-based HelixFold vs original AlphaFold2 and OpenFold?

Key findings

• HelixFold achieves competitive accuracy with AlphaFold2 on CASP14 (TM-score 0.8771 vs 0.8772) and CAMEO (TM-score 0.8885 vs 0.8862)
• End-to-end training time reduced to 7.5 days (5.3 days with hybrid parallelism) vs ~11 days for AlphaFold2 and OpenFold
• Throughput improvements in initial training: HelixFold outperforms AlphaFold2 (throughput +28.42% to +52.13%) and OpenFold; in fine-tuning, HelixFold outperforms AlphaFold2 by ~84.49%
• BP-DAP-DP hybrid parallelism yields higher efficiency than DAP alone, with lower communication overhead
• Memory optimizations (Recompute, BFloat16, in-place read/write) and tensor fusion substantially reduce peak memory and kernel launches
• Fused Gated Self-Attention reduces CPU/GPU overhead while maintaining accuracy