Transformers Discover Molecular Structure Without Graph Priors
Graph Neural Networks (GNNs) are the dominant architecture for molecular
machine learning, particularly for molecular property prediction and machine
learning interatomic potentials (MLIPs). GNNs perform message passing on
predefined graphs often induced by a fixed radius cutoff or k-nearest neighbor
scheme. While this design aligns with the locality present in many molecular
tasks, a hard-coded graph can limit expressivity due to the fixed receptive
field and slows down inference with sparse graph operations. In this work, we
investigate whether pure, unmodified Transformers trained directly on Cartesian
coordinatesx2013without predefined graphs or physical
priorsx2013can approximate molecular energies and forces. As a
starting point for our analysis, we demonstrate how to train a Transformer to
competitive energy and force mean absolute errors under a matched training
compute budget, relative to a state-of-the-art equivariant GNN on the OMol25
dataset. We discover that the Transformer learns physically consistent
patternsx2013such as attention weights that decay inversely with
interatomic distancex2013and flexibly adapts them across different
molecular environments due to the absence of hard-coded biases. The use of a
standard Transformer also unlocks predictable improvements with respect to
scaling training resources, consistent with empirical scaling laws observed in
other domains. Our results demonstrate that many favorable properties of GNNs
can emerge adaptively in Transformers, challenging the necessity of hard-coded
graph inductive biases and pointing toward standardized, scalable architectures
for molecular modeling.
68df2a3fdf49fb0df1e03bd3
Tobias Kreiman
68df2a3fdf49fb0df1e03bd4
Yutong Bai
68df2a3fdf49fb0df1e03bd5
Fadi Atieh
68df2a3fdf49fb0df1e03bd6
Elizabeth Weaver
68df2a3fdf49fb0df1e03bd7
Eric Qu
68df2a3fdf49fb0df1e03bd8
Aditi S. Krishnapriyan
Transformers Discover Molecular Structure Without Graph Priors
Graph Neural Networks (GNNs) are the dominant architecture for molecular
machine learning, particularly for molecular property prediction and machine
learning interatomic potentials (MLIPs). GNNs perform message passing on
predefined graphs often induced by a fixed radius cutoff or k-nearest neighbor
scheme. While this design aligns with the locality present in many molecular
tasks, a hard-coded graph can limit expressivity due to the fixed receptive
field and slows down inference with sparse graph operations. In this work, we
investigate whether pure, unmodified Transformers trained directly on Cartesian
coordinatesx2013without predefined graphs or physical
priorsx2013can approximate molecular energies and forces. As a
starting point for our analysis, we demonstrate how to train a Transformer to
competitive energy and force mean absolute errors under a matched training
compute budget, relative to a state-of-the-art equivariant GNN on the OMol25
dataset. We discover that the Transformer learns physically consistent
patternsx2013such as attention weights that decay inversely with
interatomic distancex2013and flexibly adapts them across different
molecular environments due to the absence of hard-coded biases. The use of a
standard Transformer also unlocks predictable improvements with respect to
scaling training resources, consistent with empirical scaling laws observed in
other domains. Our results demonstrate that many favorable properties of GNNs
can emerge adaptively in Transformers, challenging the necessity of hard-coded
graph inductive biases and pointing toward standardized, scalable architectures
for molecular modeling.
Transformers trained directly on Cartesian coordinates can achieve competitive performance in molecular energy and force prediction without predefined graphs, demonstrating adaptability and scalability.
Graph Neural Networks GNNs molecular machine learning molecular property prediction machine learning interatomic potentials MLIPs message passing predefined graphs fixed radius cutoff k-nearest neighbor receptive field inference sparse graph operations Transformers Cartesian coordinates physical priors energy and force mean absolute errors OMol25 dataset attention weights interatomic distance molecular environments hard-coded biases scaling training resources empirical scaling laws standardized architectures molecular modeling