Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge

Created by
  • Haebom

Author

Riccardo Cantini, Alessio Orsino, Massimo Ruggiero, Domenico Talia

Outline

As LLMs gain societal importance, inherent bias concerns have emerged. This study proposes a scalable benchmarking framework to assess the robustness of LLMs against adversarial bias induction. We systematically examine models across multiple tasks targeting various sociocultural biases, quantify robustness using an LLM-as-a-Judge approach, and employ jailbreaking techniques to expose security vulnerabilities. We release a curated dataset of bias-related prompts called CLEAR-Bias and identify DeepSeek V3 as the most reliable Judge LLM. Age, disability, and cross-bias are the most prominent findings. Some smaller models outperform larger models, and jailbreaking attacks are effective on all models.

Takeaways, Limitations

•
Takeaways:
◦
We present a benchmarking framework for systematically assessing bias in LLMs.
◦
Improving research accessibility by making the CLEAR-Bias dataset public.
◦
Suggesting that model safety may be more influenced by training and architecture than scale.
◦
Highlighting LLM security vulnerabilities by verifying the effectiveness of jailbreak attacks.
◦
We found that models specialized for the medical field may be less safe than general-purpose models.
•
Limitations:
◦
It may not be possible to cover all LLMs perfectly.
◦
Due to the nature of jailbreak attacks, it is difficult to guarantee complete security.
◦
May be limited to research on specific biases.
◦
Judge LLM's dependence may affect the evaluation results.
👍
Made with Slashpage