This paper addresses the issue of low-energy molecular configuration (spatial arrangement of atoms in a molecule) sampling, which is a critical task for many computations in drug discovery and optimization. While many specialized isoveginal networks have been designed to generate molecular configurations from 2D molecular graphs, anisotropic transformer models have recently emerged as an alternative due to their scalability and improved generalization performance. However, there has been a concern that anisotropic models require large model sizes to compensate for the lack of isoveginal bias. In this paper, we show that appropriately chosen positional encoding effectively addresses this size limitation. When the standard transformer model incorporating relative positional encoding for molecular graphs is extended to 25 million parameters, it outperforms the state-of-the-art anisotropic baseline model with 64 million parameters on the GEOM-DRUGS benchmark. The relative positional encoding is implemented as a negative attention bias that linearly increases with the shortest path distance between graph nodes of various gradients, similar to the widely used ALiBi technique in NLP. This architecture has the potential to serve as a foundation for a new class of molecular configuration generation models.
