Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Enhancing Generalization of Spiking Neural Networks Through Temporal Regularization

Created by
  • Haebom

Author

Boxuan Zhang, Zhen Xu, Kuan Tao

Outline

In this paper, we propose a temporal regularization training (TRT) method that introduces a time-dependent regularization mechanism to solve the overfitting problem of spiking neural networks (SNNs), which are effective for event-based neuromorphic data processing. Directly trained SNNs suffer from severe overfitting due to the limited size of neuromorphic datasets and the gradient mismatch problem, and TRT alleviates this problem by imposing stronger constraints on the early time steps. We compare the performance of TRT with state-of-the-art methods on CIFAR10/100, ImageNet100, DVS-CIFAR10, and N-Caltech101 datasets, and verify the effectiveness of TRT through ablation studies including loss landscape visualization and learning curve analysis. In addition, we present a theoretical interpretation of the temporal regularization mechanism of TRT based on the results of Fisher information analysis, and reveal the temporal information concentration (TIC) phenomenon by tracking Fisher information during TRT training. This is a phenomenon in which Fisher information is gradually concentrated in the early time steps, and it shows that the time-decay regularization mechanism of TRT improves the generalization performance of the model by inducing the network to learn strong features in the early time steps with rich information. The source code is available on GitHub.

Takeaways, Limitations

•
Takeaways:
◦
A novel training method (TRT) is presented to effectively alleviate the overfitting problem of SNNs.
◦
Emphasize the importance of early time steps and improve generalization performance through time normalization mechanism.
◦
Presentation of theoretical basis for TRT through Fisher information analysis and discovery of temporal information concentration (TIC) phenomenon
◦
Achieving state-of-the-art performance on diverse datasets and opening up source code
•
Limitations:
◦
The effectiveness of the proposed method may be limited to specific datasets and network structures. Additional validation in various environments is needed.
◦
Lack of detailed discussion on hyperparameter optimization of TRT.
◦
Further research is needed on the generalizability of the time normalization mechanism.
👍
Made with Slashpage