This is a page that curates AI-related papers published worldwide. All content here is summarized using Google Gemini and operated on a non-profit basis. Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.
Spatial Frequency Modulation for Semantic Segmentation
Created by
Haebom
Author
Linwei Chen, Ying Fu, Lin Gu, Dezhi Zheng, Jifeng Dai
Outline
This paper points out that although spatial high-frequency information (e.g., fine texture) contributes significantly to the semantic segmentation accuracy, high-frequency components may be aliased or distorted when passing through downsampling layers such as strided convolution due to the Nyquist-Shannon sampling theorem. To address this issue, we propose a novel spatial frequency modulation (SFM) technique that modulates high-frequency features to lower frequencies before downsampling and then demodulates them again during upsampling. We implement the modulation via adaptive resampling (ARS) and design a lightweight add-on that expands the signal by densely sampling the high-frequency region and then lowers the frequency according to the frequency scaling property. In addition, we propose multi-scale adaptive upsampling (MSAU) to demodulate the modulated features and recover high-frequency information via non-uniform upsampling. This module improves the segmentation by explicitly exploiting the information interaction between dense and sparsely resampled regions at multiple scales. Both modules can be seamlessly integrated with various architectures ranging from convolutional neural networks to transformers. Through feature visualization and analysis, we verify that the proposed method effectively mitigates aliasing while successfully preserving details even after demodulation. Finally, we extend the SFM to image classification, adversarial robustness, instance segmentation, and panorama segmentation tasks to verify the wide applicability and effectiveness of SFM. The code is available at https://github.com/Linwei-Chen/SFM .
A novel SFM technique is presented to effectively solve the problem of high-frequency information loss that occurs during downsampling.
◦
Reduce aliasing and distortion of high-frequency information and preserve details through adaptive resampling (ARS) and multi-scale adaptive upsampling (MSAU) modules.
◦
Ensures wide applicability through compatibility with various architectures (CNN, Transformer).
◦
Demonstrated performance improvements in a variety of tasks including image classification, adversarial robustness, instance segmentation, and panorama segmentation.
•
Limitations:
◦
Lack of detailed analysis of the computational cost and memory usage of the proposed method.
◦
Experimental results on various datasets may be limited.
◦
Lack of detailed explanation of hyperparameter optimization of ARS and MSAU modules.
