Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health

Created by
  • Haebom

Author

Nobin Sarwar, Shubhashis Roy Dipta

FedMentor: Large-Scale Language Model Adaptation for Privacy

Outline

This paper presents a study on privacy-preserving adaptation of large-scale language models (LLMs) in sensitive domains (e.g., mental health). To balance model utility, security, and strict confidentiality, we propose FedMentor, a federated fine-tuning framework that integrates Low-Rank Adaptation (LoRA) and domain-aware differential privacy (DP). FedMentor allows each client (domain) to apply a customized DP noise scale proportional to its data sensitivity, and the server adaptively reduces the noise when utility falls below a threshold. Experiments on three mental health datasets demonstrate that FedMentor improves security and reduces toxicity while maintaining utility compared to standard federated learning (FL). The framework scales to a backbone with up to 1.7 billion parameters on a single GPU client, requiring less than 173 MB of communication per round.

Takeaways, Limitations

•
Takeaways:
◦
FedMentor provides a federated learning-based framework that enhances the security and usability of LLMs while maintaining privacy.
◦
We present practical ways to securely distribute LLMs in fields that deal with sensitive data, such as mental health.
◦
Combining LoRA and DP effectively achieves a balance between model usability and privacy.
◦
Experimental results show that FedMentor increases the safe output ratio, reduces toxicity, and maintains usability close to the private baseline and centralized upper bound.
◦
Supports large-scale models in a single GPU environment with low communication costs.
•
Limitations:
◦
Only experimental results for a specific dataset (mental health-related) are presented, requiring further validation of generalizability to other sensitive domains.
◦
Further research may be needed to find the optimal balance between model performance and privacy.
◦
Further analysis is needed on the loss of usability due to the application of DP.
◦
Further research is needed on the specific interactions and optimizations of LoRA and DP.
👍
Made with Slashpage