ProGen2: Exploring the Boundaries of Protein Language Models

06/27/2022
by   Erik Nijkamp, et al.
19

Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial intelligence-driven protein design. However, we lack a sufficient understanding of how very large-scale models and data play a role in effective protein model development. We introduce a suite of protein language models, named ProGen2, that are scaled up to 6.4B parameters and trained on different sequence datasets drawn from over a billion proteins from genomic, metagenomic, and immune repertoire databases. ProGen2 models show state-of-the-art performance in capturing the distribution of observed evolutionary sequences, generating novel viable sequences, and predicting protein fitness without additional finetuning. As large model sizes and raw numbers of protein sequences continue to become more widely accessible, our results suggest that a growing emphasis needs to be placed on the data distribution provided to a protein sequence model. We release the ProGen2 models and code at https://github.com/salesforce/progen.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/29/2023

ProtFIM: Fill-in-Middle Protein Sequence Design via Protein Language Models

Protein language models (pLMs), pre-trained via causal language modeling...
research
01/09/2022

λ-Scaled-Attention: A Novel Fast Attention Mechanism for Efficient Modeling of Protein Sequences

Attention-based deep networks have been successfully applied on textual ...
research
07/13/2020

ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing

Computational biology and bioinformatics provide vast data gold-mines fr...
research
11/19/2019

PDBMine: A Reformulation of the Protein Data Bank to Facilitate Structural Data Mining

Large scale initiatives such as the Human Genome Project, Structural Gen...
research
07/17/2023

Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge

Understanding protein interactions and pathway knowledge is crucial for ...
research
01/24/2022

Guided Generative Protein Design using Regularized Transformers

The development of powerful natural language models have increased the a...
research
05/18/2023

Vaxformer: Antigenicity-controlled Transformer for Vaccine Design Against SARS-CoV-2

The SARS-CoV-2 pandemic has emphasised the importance of developing a un...

Please sign up or login with your details

Forgot password? Click here to reset