Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models

Created by
  • Haebom

Author

Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Praneeth Vepakomma

Outline

This paper presents ABBA, a parameter-efficient fine-tuning (PEFT) methodology for efficiently adapting large-scale language models (LLMs) to new domains. ABBA is a novel PEFT architecture that reparameterizes updates using the Hadamard product of pre-trained weights. Unlike existing PEFT methods, ABBA completely decouples updates from the pre-trained weights, allowing both components to be optimized independently. This enables higher expressive power with the same parameter budget and outperforms existing PEFT methods on arithmetic and common-sense reasoning benchmarks.

Takeaways, Limitations

•
Takeaways:
◦
ABBA presents a novel approach to PEFT methodology to improve the efficiency of domain adaptation in LLM.
◦
It outperforms existing PEFT methodologies on arithmetic and common-sense reasoning benchmarks.
◦
We improved expressiveness by separating updates from pre-trained weights.
◦
We increased the reproducibility of our research by making our code publicly available via GitHub.
•
Limitations:
◦
Although not explicitly stated in the paper, further analysis may be needed, including comparative analysis with other PEFT methodologies, evaluation of generalization performance for different models and tasks, and further analysis of computational cost and memory usage.
👍
Made with Slashpage