To overcome the limitations of existing spectrum recognition methods, which exhibit limited generalization and suboptimal accuracy across diverse spectral environments and tasks, we propose SpectrumFM, a spectrum-based model. SpectrumFM effectively captures fine-grained local signal structures and high-dimensional global dependencies in spectral data through an innovative spectral encoder that leverages convolutional neural networks and a multi-head self-attention mechanism. To enhance the model's adaptability, we develop two novel self-supervised learning tasks—mask reconstruction and next-slot signal prediction—to pretrain SpectrumFM and learn rich, transferable representations. Furthermore, we leverage low-rank adaptation (LoRA) parameter-efficient fine-tuning to enable SpectrumFM to seamlessly adapt to diverse subspectral recognition tasks, including spectrum sensing (SS), anomaly detection (AD), and wireless technology classification (WTC). Extensive experiments demonstrate that SpectrumFM outperforms state-of-the-art methods, specifically improving detection probability by 30% at -4 dB SNR in the SS task, improving area under the curve (AUC) by over 10% in the AD task, and improving WTC accuracy by 9.6%.