[Paper Review] ProGen: Language Modeling for Protein Generation

Ali Madani, Bryan McCann+6 more|arXiv (Cornell University)|Mar 8, 2020

Genomics and Phylogenetic Studies35 references34 citations

TL;DR

ProGen is a 1.2B-parameter conditional language model trained on ~281M protein sequences with conditioning tags, enabling controllable protein generation that aligns with structure- and function-related metrics, including zero-shot cases and fine-tuning on unseen families.

ABSTRACT

Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ~280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.

Motivation & Objective

• Leverage large-scale protein sequence data to enable unsupervised, generative protein design.
• Develop a conditional Transformer model that can be steered by taxonomy and functional tags.
• Evaluate generated proteins across sequence, secondary structure, and conformational energy metrics.
• Demonstrate capabilities on unseen protein families and zero-shot functional selection.
• Explore practical applications in directed evolution and de novo design.

Proposed method

• Train a Transformer-based conditional language model with 36 layers, 1028-dim embeddings, and 8 attention heads per layer to model p(x) where x combines conditioning tags and amino acid tokens.
• Condition generation on a rich set of tags (1100 keyword tags and 100k taxonomic terms) appended to sequences, enabling controllable protein design.
• Use a dataset of ~281M protein sequences from Uniparc/UniProt/SWI... with sequence length capped at 512 tokens, plus reverse sequences, and dropout on conditioning tags.
• Sample next tokens during generation with top-k sampling and a repetition penalty to encourage diversity while preserving structure.
• Evaluate using perplexity, hard and soft (BLOSUM62) token accuracies, and higher-level metrics: primary sequence similarity, secondary structure accuracy (PSIPRED with MSAs), and conformational energy (Rosetta-RelaxBB).
• Compare against uniform and empirical baselines; fine-tune on unseen families to improve out-of-domain generalization.

Experimental results

Research questions

• RQ1 Can a large conditional language model learn the distribution of evolutionarily viable protein sequences from large, annotated sequence databases?
• RQ2 Do conditioning tags improve controllable protein generation across sequence and higher-order structural properties?
• RQ3 How well does ProGen generalize to unseen protein families (OOD) and can fine-tuning improve this generalization?
• RQ4 Do ProGen-generated sequences exhibit native-like secondary structure and favorable conformational energies after threading/relaxation?
• RQ5 Can ProGen perform zero-shot selection to identify high-fitness protein variants (e.g., GB1) without supervised training?

Key findings

• ProGen significantly outperforms uniform and empirical baselines in perplexity and hard accuracy, including on held-out protein families.
• Perplexity decreases and per-token accuracy increases with longer amino acid context and more conditioning tags.
• Generated sequences show improved secondary structure accuracy and conformational energies close to native proteins, often better than baselines after relaxation.
• Fine-tuning ProGen on unseen protein families yields substantial gains over training from random initialization, especially in OOD settings.
• VEGFR2 completion samples preserve key structural elements and maintain near-native energy levels across generation lengths.
• In zero-shot GB1 fitness selection, ProGen identifies high-fitness variants without supervised training, outperforming random mutation baselines.