This paper presents a framework called Graph-Aligned Large Language Models (GALLa). GALLa represents structural information (e.g., data flow) of code in a graph format, providing additional information to the text-token-based learning methods of existing code language models (LLMs). To overcome the scalability limitations of existing models that utilize structural information due to the need for modifications to the Transformer architecture, GALLa leverages graph neural networks (GNNs) and cross-modal alignment techniques to inject structural information as an auxiliary task during the fine-tuning process. This framework is model- and task-independent, making it applicable to a variety of code LLMs and subtasks. It requires structural graph data only during training and incurs no additional overhead during inference. Experiments on five code tasks using seven LLMs with parameters ranging from 350 million to 14 billion demonstrate that GALLa outperforms the baseline model even on powerful models such as LLaMA3 and Qwen2.5-Coder.
