This paper addresses the problem that graph-based methods developed for brain disease diagnosis using functional connectivity rely on existing brain atlases and overlook the mixing effects of rich information within the atlas and site- and phenotypic variability. To this end, we propose a two-stage Brain-to-Population Graph Learning (B2P-GL) framework that integrates semantic similarity of brain regions with state-based population graph modeling. In the first stage, we enrich the graph representation by leveraging brain atlas knowledge from GPT-4 and refine the brain graph using an adaptive node reassignment graph attention network. In the second stage, we integrate phenotypic data into the population graph construction and feature fusion to mitigate mixing effects and improve diagnostic performance. Experimental results on the ABIDE I, ADHD-200, and Rest-meta-MDD datasets demonstrate that B2P-GL outperforms state-of-the-art methods in predictive accuracy and enhances interpretability.