Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Social Debiasing for Fair Multi-modal LLMs

Created by
  • Haebom

Author

Harry Cheng, Yangyang Guo, Qingpei Guo, Ming Yang, Tian Gan, Weili Guan, Liqiang Nie

Outline

This paper presents two major contributions to address the problem of social bias in multimodal large-scale language models (MLLMs). First, we introduce the Comprehensive Counterfactual Dataset (CMSC), which includes 18 diverse and balanced social concepts. CMSC complements existing datasets, enabling a more comprehensive approach to social bias mitigation. Second, we propose a counter-stereotype debiasing (CSD) strategy to mitigate social bias in MLLMs by leveraging the counter-concept of widespread stereotypes. CSD integrates a novel bias-aware data sampling method and loss rebalancing to enhance the model's bias reduction efficiency. Extensive experiments using four major MLLM architectures demonstrate that the CMSC dataset and CSD strategy effectively reduce social bias compared to existing methods, achieving this without compromising overall performance on common multimodal inference benchmarks.

Takeaways, Limitations

•
Takeaways:
◦
Contributes to solving the social bias problem in MLLM by providing a new counterfactual dataset (CMSC) that includes diverse and balanced social concepts.
◦
We present a novel counter-stereotype debiasing (CSD) strategy that reduces social bias in MLLM more effectively than existing methods.
◦
The CSD strategy achieves bias reduction without compromising the general multimodal inference performance.
◦
We validate the effectiveness of CMSC and CSD through extensive experiments on various MLLM architectures.
•
Limitations:
◦
The CMSC dataset's coverage of social concepts may not be fully comprehensive. It needs to be expanded to include a wider range of social concepts.
◦
The effectiveness of the CSD strategy may vary depending on the specific MLLM architecture and training data used. Further research is needed on other models and data.
◦
Further research may be needed to define and measure social bias. Various measurement indicators and assessment methodologies should be considered.
👍
Made with Slashpage