This paper introduces Velocity-Regularized Adam (VRAdam), a novel optimizer for training deep learning models. Inspired by the fourth-order term in kinetic energy, VRAdam automatically reduces the learning rate as weight updates increase, thereby enhancing training stability. Combined with Adam's parameter-specific scaling, VRAdam forms a powerful hybrid optimizer. VRAdam provides a theoretical analysis of stability bounds from a physics and control perspective, and derives a convergence bound of $\mathcal{O}(\ln(N)/\sqrt{N})$ for a stochastic nonconvex objective function under moderate assumptions. Using a variety of architectures and training methods (e.g., CNN, Transformer, GFlowNet), it outperforms existing optimizers, including AdamW, on a variety of tasks such as image classification, language modeling, and generative modeling.
