This paper proposes GuARD, a novel model for anomaly detection in text-rich graphs. Existing large-scale language model (LLM)-based anomaly detection methods suffer from limitations, such as inability to effectively utilize textual information or failure to consider structural features of the graph. GuARD addresses these challenges by combining the structural features of graph-based methods with fine-grained semantic properties extracted from small-scale language models. It utilizes an advanced multimodal, multi-pass directive adjustment framework, optimized to integrate both textual and structural modalities. Experimental results on four datasets demonstrate superior performance, training speed, and inference speed compared to existing methods.