MaCP (Minimal yet Mighty Adaptive Cosine Projection) is a novel adaptive method for fine-tuning large-scale base models. It achieves superior performance while using fewer parameters and memory compared to existing methods. Its core idea is to leverage the superior energy compression and de-correlation properties of cosine projection to improve model efficiency and accuracy. It projects the weight changes from low-dimensional adaptation into the discrete cosine space, partitions the weight changes across multiple levels of the discrete cosine spectrum, and selects the most significant frequency components from each partition. It demonstrates effectiveness across a wide range of tasks, including unimodal tasks such as natural language understanding, natural language generation, and text summarization, as well as multimodal tasks such as image classification and video understanding. It offers higher accuracy and significantly lower computational complexity and memory requirements than existing methods.
