GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text

08/14/2023
by   Pengfei Liu, et al.
0

Large language models have made significant strides in natural language processing, paving the way for innovative applications including molecular representation and generation. However, most existing single-modality approaches cannot capture the abundant and complex information in molecular data. Here, we introduce GIT-Mol, a multi-modal large language model that integrates the structure Graph, Image, and Text information, including the Simplified Molecular Input Line Entry System (SMILES) and molecular captions. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture capable of mapping all modalities into a unified latent space. Our study develops an innovative any-to-language molecular translation strategy and achieves a 10 captioning, a 5 in molecule generation validity compared to baseline or single-modality models.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset