Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

IPA: An Information-Preserving Input Projection Framework for Efficient Foundation Model Adaptation

Created by
  • Haebom

Author

Yuan Yin, Shashanka Venkataramanan, Tuan-Hung Vu, Andrei Bursuc, Matthieu Cord

Outline

This paper proposes Information-Preserving Adaptation (IPA), a feature-aware projection framework, to address the Limitations of parameter-efficient fine-tuning (PEFT) methods such as LoRA. While LoRA uses randomly initialized dimensionality reduction, which incurs information loss, IPA explicitly preserves information in the reduced hidden space through an algorithm that approximates the main principal components. In linear cases, IPA enables efficient projector pretraining with negligible inference overhead.

Takeaways, Limitations

•
Takeaways:
◦
Achieve performance improvements by overcoming the limitations of LoRA's random dimensionality reduction.
◦
It shows an average accuracy improvement of 1.5 and 2.3 points over LoRA and DoRA on benchmarks such as Commonsense reasoning and VTAB-1k.
◦
When the projection is fixed, it achieves performance equivalent to LoRA with about half the learnable parameters of LoRA.
◦
Minimize inference overhead through efficient projector pre-training.
•
Limitations:
◦
Currently, only the IPA algorithm for linear cases has been presented. Extensions to nonlinear cases are needed.
◦
Further research is needed on generalization performance on tasks other than the presented benchmarks.
◦
The efficiency and performance improvements of IPA may vary depending on the dataset and model used.
👍
Made with Slashpage