In this paper, we propose a novel method, Cross-Task Alignment (CTA), to improve test-time training (TTT). Unlike existing TTT methods, CTA does not require a special model architecture, and aligns supervised learning encoders with self-supervised learning encoders, inspired by the success of multi-modal contrastive learning. This process strengthens the alignment between the learned representations of the two models, which mitigates the risk of gradient interference, preserves the intrinsic robustness of self-supervised learning, and enables more meaningful updates at test time. Experimental results show that it significantly improves the robustness and generalization performance compared to the state-of-the-art on several benchmark datasets.
