MolXPT: Wrapping Molecules with Text for Generative Pre-training
Generative pre-trained Transformer (GPT) has demonstrates its great success
in natural language processing and related techniques have been adapted into
molecular modeling. Considering that text is the most important record for
scientific discovery, in this paper, we propose MolXPT, a unified language
model of text and molecules pre-trained on SMILES (a sequence representation of
molecules) wrapped by text. Briefly, we detect the molecule names in each
sequence and replace them to the corresponding SMILES. In this way, the SMILES
could leverage the information from surrounding text, and vice versa. The above
wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem
are all fed into a language model for pre-training. Experimental results
demonstrate that MolXPT outperforms strong baselines of molecular property
prediction on MoleculeNet, performs comparably to the best model in
text-molecule translation while using less than half of its parameters, and
enables zero-shot molecular generation without finetuning.
6466faedcdea48159aa5af04
Zequn Liu
6466faedcdea48159aa5af05
Wei Zhang
6466faedcdea48159aa5af06
Yingce Xia
claimed_verified
2023-05-19T19:04:39.990Z
644a2eddd9a3ae834104c654
/avatars/a52dbbd6ef1dca5ba9a94d0f564204de.svg
Yingce Xia
user
xyc1207
6466faedcdea48159aa5af07
Lijun Wu
6466faedcdea48159aa5af08
Shufang Xie
6466faedcdea48159aa5af09
Tao Qin
6466faedcdea48159aa5af0a
Ming Zhang
6466faedcdea48159aa5af0b
Tie-Yan Liu
MolXPT: Wrapping Molecules with Text for Generative Pre-training
Generative pre-trained Transformer (GPT) has demonstrates its great success
in natural language processing and related techniques have been adapted into
molecular modeling. Considering that text is the most important record for
scientific discovery, in this paper, we propose MolXPT, a unified language
model of text and molecules pre-trained on SMILES (a sequence representation of
molecules) wrapped by text. Briefly, we detect the molecule names in each
sequence and replace them to the corresponding SMILES. In this way, the SMILES
could leverage the information from surrounding text, and vice versa. The above
wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem
are all fed into a language model for pre-training. Experimental results
demonstrate that MolXPT outperforms strong baselines of molecular property
prediction on MoleculeNet, performs comparably to the best model in
text-molecule translation while using less than half of its parameters, and
enables zero-shot molecular generation without finetuning.
MolXPT, a unified text and molecular language model pre-trained on wrapped SMILES and text data, outperforms baselines in molecular property prediction, matches top models in text-molecule translation with fewer parameters, and supports zero-shot molecular generation.
Generative pre-trained Transformer MolXPT SMILES language model PubMed PubChem molecular property prediction MoleculeNet text-molecule translation zero-shot molecular generation