This is a page that curates AI-related papers published worldwide. All content here is summarized using Google Gemini and operated on a non-profit basis. Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.
In this paper, we propose a lightweight framework, Spectral Decoupled Knowledge Distillation (SDKD), to address the low training efficiency and high memory consumption of complex models in spatiotemporal prediction tasks such as traffic flow, combustion dynamics, and weather forecasting. SDKD transfers the multi-scale spatiotemporal representation of a complex teacher model to an efficient lightweight student network. The teacher model follows the encoder-latent evolution-decoder architecture, and the latent evolution module uses convolution and Transformer (global low-frequency modeler) to separate high-frequency details and low-frequency trends. However, the multi-layer convolution and deconvolution architectures lead to slow training and high memory usage. To address this issue, we propose a frequency-aligned knowledge distillation strategy that extracts multi-scale spectral features from the latent space of the teacher model, which includes both high- and low-frequency components, and guides the lightweight student model to capture both local micro-changes and global evolution patterns. Experimental results show that SDKD significantly improves performance on the Navier-Stokes equations dataset, achieving up to 81.3% MSE reduction and up to 52.3% MAE reduction, effectively capturing high-frequency changes and long-term trends while reducing computational complexity. The source code can be found at https://github.com/itsnotacie/SDKD .
Proposing a lightweight framework SDKD that effectively addresses training efficiency and memory consumption issues for complex spatiotemporal prediction models.
◦
Effectively learn both high-frequency changes and long-term trends through a frequency-aligned knowledge distillation strategy.
◦
Experimentally verify performance improvement through MSE and MAE reduction on the Navier-Stokes equations dataset.
•
Limitations:
◦
Additional experiments are needed to evaluate the generalization performance of the proposed method. Further performance evaluations on various spatiotemporal prediction tasks are needed.
◦
Since only results for a specific dataset are presented, applicability to other types of datasets needs to be verified.
◦
Lack of detailed description of the teacher model design and hyperparameter settings. Additional information is needed to ensure reproducibility.
