This paper addresses the problem of integrating structured graph data with rich textual information of nodes, especially for heterogeneous node classification. Existing approaches are either computationally expensive or struggle to effectively fuse different modalities. In this paper, we propose a novel architecture, the Graph Mask Language Model (GMLM), that efficiently combines graph neural networks (GNNs) and pre-trained language models (PLMs). GMLM introduces three key innovations: (i) a dynamic active node selection strategy for scalable PLM text processing, (ii) a GNN-specific contrastive pre-training step utilizing soft masking with learnable graph [MASK] tokens, and (iii) a dedicated fusion module that integrates RGCN-based GNN embeddings with PLM (GTE-Small & DistilBERT) embeddings. We demonstrate the superiority of GMLM through extensive experiments on heterogeneous benchmarks (Cornell, Wisconsin, Texas). In particular, GMLM (DistilBERT) achieves significant performance improvements over the previous state-of-the-art baseline models, with an accuracy improvement of more than 4.7% at Cornell and more than 2.0% at Texas. This study highlights the benefits of target-oriented PLM engagement and modality-specific pretraining for efficient and improved learning in text-rich graphs.
