This page organizes papers related to artificial intelligence published around the world. This page is summarized using Google Gemini and is operated on a non-profit basis. The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.
Accurate and Efficient Low-Rank Model Merging in Core Space
Created by
Haebom
Author
Aniello Panariello, Daniel Marczak, Simone Magistri, Angelo Porrello, Bart{\l}omiej Twardowski, Andrew D. Bagdanov, Simone Calderara, Joost van de Weijer
Outline
This paper addresses the challenges associated with merging low-rank adaptation (LoRA) for large-scale neural networks. The rise of parameter-efficient adaptation techniques like LoRA has made model fine-tuning easier. While model fine-tuning using LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging full-size weight matrices. This paper proposes a Core Space merging framework that merges LoRA-enabled models within a common alignment base, significantly improving accuracy across tasks while maintaining the efficiency of low-rank adaptation. We also provide a formal proof that projection into Core Space guarantees information loss and provide a complexity analysis demonstrating its efficiency benefits. Extensive experimental results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks, while consuming only a fraction of the computational resources.
Takeaways, Limitations
•
Takeaways:
◦
Improved accuracy while maintaining the efficiency of LoRA-based model merging.
◦
Proposal of a Core Space merge framework.
◦
We present a formal proof that projections into Core Space guarantee information loss.
◦
Achieving superior performance over existing technologies in vision and language tasks.
◦
Reduced computational resource usage.
•
Limitations:
◦
No mention of Limitations in the paper itself (not possible to determine from the Abstract alone)
