This paper introduces SpectralEarth, a large-scale, multi-perspectral hyperspectral image dataset leveraging data from the Environmental Mapping and Analysis Program (EnMAP). SpectralEarth comprises 538,974 image patches (415,153 unique locations) collected from 11,636 globally distributed EnMAP scenes, 17.5% of which contain multiple time stamps, enabling multi-perspectral analysis. In this paper, we pretrain hyperspectral-based models on SpectralEarth using state-of-the-art self-supervised learning algorithms and integrate a spectral adapter into an existing vision backbone to accommodate the unique characteristics of HSI. Furthermore, we build nine downstream datasets for land cover, crop type mapping, and tree species classification to provide benchmarks for model evaluation. Experimental results demonstrate the model's versatility and generalization performance across a variety of tasks and sensors, highlighting its computational efficiency during model fine-tuning.