Despite recent advances in image-based visual saliency prediction, predicting visual fixations across multiple datasets remains challenging, and we demonstrate that this is due to dataset bias. Models trained on one dataset significantly degrade when applied to other datasets. Increasing dataset diversity does not resolve this inter-dataset gap, with approximately 60% of the gap attributed to dataset-specific bias. To address this generalization gap, we propose a novel architecture based on a dataset-independent encoder-decoder architecture, adding fewer than 20 dataset-specific parameters that control interpretable mechanisms such as multiscale architecture, center bias, and fixation spread. Adapting these parameters alone to new data addresses over 75% of the generalization gap, achieving significant improvements with as few as 50 samples. The proposed model achieves new state-of-the-art performance on three datasets from the MIT/Tübingen Saliency Benchmark (MIT300, CAT2000, and COCO-Freeview) and also demonstrates excellent generalization performance on unrelated datasets. Additionally, the model exhibits complex multi-scale effects combining both absolute and relative magnitude, providing valuable insights into spatial visual significance properties.
