This paper proposes DispFormer, a novel method for estimating subsurface shear wave velocity (Vs) profiles using surface wave dispersion curve inversion. DispFormer uses a Transformer-based neural network to invert Vs profiles from Rayleigh wave phases and group dispersion curves, addressing the computational cost, non-uniformity, and initial model sensitivity of existing methods. It processes each cycle independently, enabling data of varying lengths, and employs a training strategy that considers depth sensitivity. Pretrained on synthetic data, DispFormer demonstrates superior performance on real data through both zero-shot and few-shot learning strategies, achieving lower data residuals than existing methods. This demonstrates the potential of deep learning, which integrates physical information, for geophysical applications.