In this paper, we study a knowledge transfer method between time series modalities using a multimodal temporal representation space for human activity recognition (HAR) to leverage the potential of various sensors. In particular, we explore the setting where the modalities used for testing are unlabeled during training (Unsupervised Modality Adaptation (UMA). Existing UMA approaches are categorized into Student-Teacher or Contrastive Alignment methods, and point out the limitation that these methods generally compress continuous time data samples into a single latent vector, which hinders the ability to transfer temporal information through real-world temporal distortions. To address this, we propose Cross-modal Transfer Through Time (C3T), which preserves temporal information to better handle dynamic sensor data. C3T achieves this by aligning a series of temporal latent vectors across sensing modalities. Through extensive experiments on various camera+IMU datasets, we show that C3T outperforms existing UMA methods by at least 8% and exhibits excellent robustness against temporal distortions such as time shift, misalignment, and dilation. These results suggest that C3T has significant potential for developing generalizable models for time-series sensor data, opening up new avenues for various multi-modal applications.
