[공지사항]을 빙자한 안부와 근황 
Show more

Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

VectorFit: Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models

Created by
  • Haebom

Author

Suhas G Hegde, Shilpy Kaur, Aruna Tiwari

Outline

This paper points out the problem that existing Parameter-Efficient Fine-Tuning (PEFT) methods have limitations in improving performance because they freeze pre-learned weights and learn new low-rank or sparse weights. Since existing methods learn new weights from scratch, performance degradation occurs. VectorFit proposes a new parameterization method that adaptively learns the singular vectors and biases of W by utilizing the information inherent in the pre-learned weights W. Through this, we achieve performance similar to full fine-tuning with far fewer parameters (9x reduction) than existing PEFT methods, and experimentally demonstrate that our method performs superiorly in various tasks such as natural language understanding and generation, question answering, and image classification and generation through 19 datasets.

Takeaways, Limitations

•
Takeaways:
◦
We present a novel PEFT method, VectorFit, which effectively utilizes information from pre-learned weights to maximize parameter efficiency.
◦
It achieves high performance with significantly fewer parameters than existing PEFT methods, demonstrating its effectiveness even in environments with limited computing resources.
◦
It demonstrates excellent performance in various language and vision tasks, suggesting wide application potential.
•
Limitations:
◦
It is difficult to conclude that the 19 datasets presented in the paper fully guarantee the generalization performance of VectorFit. Additional experiments on more diverse and extensive datasets may be necessary.
◦
We cannot rule out the possibility that the performance improvements of VectorFit are limited to certain types of models or tasks. Further research is needed on various model architectures and tasks.
👍
Made with Slashpage