Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Fairness Evaluation of Large Language Models in Academic Library Reference Services

Created by
  • Haebom

Author

Haining Wang, Jason Clark, Yueru Yan, Star Bradley, Ruiyang Chen, Yiqiong Zhang, Hengyi Fu, Zuoyu Tian

Outline

This paper evaluates whether large-scale language models (LLMs) can serve all users fairly, regardless of demographic characteristics or social status, in a library that leverages virtual reference services. Using six state-of-the-art LLMs, we evaluate whether LLMs discriminate in their responses based on user identity by prompting users to help them in different genders, races/ethnicities, and institutional roles. We find no discrimination based on race or ethnicity, and only a small stereotyped bias toward women in one model. LLMs show subtle adaptations to institutional roles through linguistic choices related to formality, politeness, and domain-specific vocabulary, which reflect professional norms rather than discriminatory treatment.

Takeaways, Limitations

Takeaways: The current LLM suggests that the academic library reference service is reasonably well prepared to support fair and contextual communication. No discrimination was found based on race or ethnicity, and gender bias was minimal. The LLM appears to use appropriate language according to the institutional role.
Limitations: The limited number of models examined, only six, and the finding of some stereotypical bias against women in certain models suggests the need for further research. Additional research is needed to more broadly evaluate LLM responses to a variety of question types or situations.
👍