This study aims to improve the predictive accuracy and clinical interpretability of an emergency department (ED) mental health readmission risk prediction model by integrating large-scale language models (LLMs) with machine learning. We analyzed 42,464 ED visit records of 27,904 unique mental health patients from January 2018 to December 2022 to evaluate the predictive accuracy of ED readmission within 30 days and the model interpretability using the LLM-enhanced framework. Using LLaMA 3 (8B), we achieved higher accuracy (0.882) and F1-score (0.86) than the baseline model, and also showed high accuracy (0.95) and F1-score (0.96) for SDoH classification. In particular, the LLM-based interpretability framework achieved 99% accuracy in converting model predictions into clinically relevant explanations. The LLM extraction feature improved the AUC of XGBoost from 0.74 to 0.76 and the AUC-PR from 0.58 to 0.61. In conclusion, the integration of LLM and machine learning models slightly improved the prediction accuracy while significantly improving the interpretability through automated clinically relevant explanations.
