This paper proposes the Temporal Attention Module (TAM) to improve the accuracy of cardiac anatomy segmentation. Existing cardiac segmentation methods that utilize cardiac motion information suffer from limitations such as high computational costs and poor robustness. TAM is designed as a lightweight, multi-head cross-temporal attention module that can be easily applied to various networks (CNN-based, Transformer-based, or hybrid) used in various cardiac imaging systems (2D and 3D ultrasound and MRI). Experiments on various datasets demonstrate that TAM outperforms existing methods while maintaining computational efficiency. TAM provides a robust and generalized solution that scales from 2D to 3D.