Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classification

Created by
  • Haebom

Author

Aarush Sinha

Outline

This paper addresses the problem of integrating structured graph data with rich textual information of nodes, especially for heterogeneous node classification. Existing approaches are either computationally expensive or struggle to effectively fuse different modalities. In this paper, we propose a novel architecture, the Graph Mask Language Model (GMLM), that efficiently combines graph neural networks (GNNs) and pre-trained language models (PLMs). GMLM introduces three key innovations: (i) a dynamic active node selection strategy for scalable PLM text processing, (ii) a GNN-specific contrastive pre-training step utilizing soft masking with learnable graph [MASK] tokens, and (iii) a dedicated fusion module that integrates RGCN-based GNN embeddings with PLM (GTE-Small & DistilBERT) embeddings. We demonstrate the superiority of GMLM through extensive experiments on heterogeneous benchmarks (Cornell, Wisconsin, Texas). In particular, GMLM (DistilBERT) achieves significant performance improvements over the previous state-of-the-art baseline models, with an accuracy improvement of more than 4.7% at Cornell and more than 2.0% at Texas. This study highlights the benefits of target-oriented PLM engagement and modality-specific pretraining for efficient and improved learning in text-rich graphs.

Takeaways, Limitations

Takeaways:
We present a novel architecture (GMLM) that effectively integrates GNN and PLM to achieve superior performance on heterogeneous node classification problems.
Reduce computational cost and effectively fuse various modalities through dynamic active node selection strategy, GNN-specific contrastive pre-training, and dedicated fusion module.
It shows significant performance improvement (up to 4.7% and 2.0%, respectively) over the previous state-of-the-art baseline models on the Cornell and Texas datasets.
Emphasizes the importance of goal-oriented PLM engagement and modality-specific pre-training.
Limitations:
Further research is needed on the generalization performance of the proposed model.
Applicability and performance evaluation for other types of graph data or node classification problems are needed.
Performance improvement figures for the Wisconsin dataset are not explicitly presented.
Lack of performance comparison analysis for other PLMs other than the specific PLM used (GTE-Small & DistilBERT).
👍