This paper presents a comprehensive and unbiased benchmarking of the efficiency, memory consumption, and effectiveness of spectral-based graph neural networks (spectral GNNs). This approach addresses the challenges of selecting appropriate spectral models for specific graph data and deploying them on large-scale web-scale graphs, which have arisen due to the diverse model designs and learning settings of previous studies. In this paper, we analyze and categorize 35 GNNs and 27 filters as spectral graph filters and implement them within a unified, spectral-centric framework, enabling deployment of spectral GNNs on million-scale graphs and diverse tasks. Through evaluations across various graph scales, we provide new observations and practical guidance on their effectiveness and efficiency. We also illuminate the complexities of spectral graph filter effectiveness and efficiency, and suggest potential performance enhancements through tailored spectral manipulation.