Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning

Created by
  • Haebom

Author

Le-Trung Nguyen, Ael Quelennec, Van-Tam Nguyen, Enzo Tartaglione

Outline

This paper proposes a novel short-path approach to solve the memory and computation constraints of on-device learning. Based on the existing low-rank decomposition method, we propose a method to reduce the memory usage and total training FLOPs by alleviating the activation memory bottleneck in the backpropagation process. The experimental results show that the activation memory usage is reduced by up to 120.09 times and the FLOPs are reduced by up to 1.86 times compared to the existing methods. This suggests that the efficiency of on-device learning can be greatly improved.

Takeaways, Limitations

•
Takeaways:
◦
We present a novel method that can contribute to improving the memory and computational efficiency of on-device learning.
◦
Experimentally demonstrated that it can dramatically reduce activation memory usage and FLOPs.
◦
Presenting the possibility of developing a low-power, low-latency, and high-efficiency on-device AI system.
•
Limitations:
◦
Further studies are needed to investigate the generalization performance of the proposed method and its applicability to various models/datasets.
◦
Only results for specific benchmarks are presented, so performance in other environments requires further verification.
◦
A more in-depth analysis is needed to address the potential accuracy degradation of the short-path approach.
👍
Made with Slashpage