In this paper, we propose a comprehensive resource called FalseReject, which consists of 16,000 seemingly harmful questions and structured responses across 44 safety-related categories to address the problem of safety-aligned approaches to large-scale language models (LLMs) excessively rejecting even innocent questions. We present a graph-based adversarial multi-agent interaction framework to generate diverse and complex prompts, and provide structured responses with explicit inferences to help the model accurately distinguish between safe and unsafe contexts. FalseReject includes custom training datasets and human-annotated benchmark test sets for both standard directive-tuned models and inference-driven models. We demonstrate the persistent problem of excessive rejections through extensive benchmarking on 29 state-of-the-art (SOTA) LLMs, and experimentally demonstrate that fine-tuning supervised learning using FalseReject significantly reduces unnecessary rejections without compromising overall safety or general language functions.
