Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

An empirical study of task and feature correlations in the reuse of pre-trained models

Created by
  • Haebom

Author

Jama Hussein Mohamud, Willie Brink

Outline

This paper presents a setup to experimentally study the success factors of reuse of pre-trained neural networks. The experimental results show that the correlation between two tasks plays an important role in the success of reuse of pre-trained models. The higher the correlation between tasks, the higher the success rate of reuse. Even if there is no correlation, good performance can be obtained by chance depending on the choice of pre-trained networks and optimizers. When the correlation between tasks is low, it is advantageous to reuse only the lower layers, and the number of layers to be retrained can be an indicator of the correlation between tasks and features. Finally, we show that in real scenarios, if there is a semantic correlation between two tasks, pre-trained networks can be effectively reused.

Takeaways, Limitations

•
Takeaways:
◦
We find that the success of reusing pre-trained models depends heavily on the correlation between the two tasks.
◦
We show that even when the correlation between tasks is low, unexpectedly high performance can be achieved depending on the choice of pre-training process (network, optimizer).
◦
The number of layers to reuse can be an indicator of the correlation between tasks and features.
◦
We experimentally demonstrate that reusing pre-trained networks is effective across tasks with semantic correlations.
•
Limitations:
◦
Further research is needed to determine the generalizability of the presented experimental setup.
◦
More extensive experimentation with different types of tasks and networks is needed.
◦
Further research is needed on how to quantitatively measure correlations between tasks and features.
👍
Made with Slashpage