ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

10/19/2020
by   Seyone Chithrananda, et al.
13

GNNs and chemical fingerprints are the predominant approaches to representing molecules for property prediction. However, in NLP, transformers have become the de-facto standard for representation learning thanks to their strong downstream task transfer. In parallel, the software ecosystem around transformers is maturing rapidly, with libraries like HuggingFace and BertViz enabling streamlined training and introspection. In this work, we make one of the first attempts to systematically evaluate transformers on molecular property prediction tasks via our ChemBERTa model. ChemBERTa scales well with pretraining dataset size, offering competitive downstream performance on MoleculeNet and useful attention-based visualization modalities. Our results suggest that transformers offer a promising avenue of future work for molecular representation learning and property prediction. To facilitate these efforts, we release a curated dataset of 77M SMILES from PubChem suitable for large-scale self-supervised pretraining.

READ FULL TEXT

page 1

page 2

page 3

page 4

09/05/2022

ChemBERTa-2: Towards Chemical Foundation Models

Large pretrained models such as GPT-3 have had tremendous impact on mode...
12/26/2021

Stepping Back to SMILES Transformers for Fast Molecular Representation Inference

In the intersection of molecular science and deep learning, tasks like v...
09/26/2022

Taking a Respite from Representation Learning for Molecular Property Prediction

Artificial intelligence (AI) has been widely applied in drug discovery w...
06/28/2021

LiteGEM: Lite Geometry Enhanced Molecular Representation Learning for Quantum Property Prediction

In this report, we (SuperHelix team) present our solution to KDD Cup 202...
06/03/2021

When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations

Vision Transformers (ViTs) and MLPs signal further efforts on replacing ...
02/19/2020

Molecule Attention Transformer

Designing a single neural network architecture that performs competitive...
08/23/2021

C5T5: Controllable Generation of Organic Molecules with Transformers

Methods for designing organic materials with desired properties have hig...

Code Repositories

bert-loves-chemistry

bert-loves-chemistry: a repository of HuggingFace models applied on chemical SMILES data for drug design, chemical modelling, etc.


view repo

Compound-Protein-Interaction

None


view repo