In this paper, we present a method for learning temporal causal representations, which is useful for revealing complex patterns in observational studies, to apply them to high-dimensional irregular tensor-shaped data. We extract meaningful clusters through irregular tensor decomposition and propose a new causal formulation based on them. A novel learning framework, CaRTeD, integrates temporal causal representation learning and irregular tensor decomposition to provide a blueprint for downstream tasks such as latent structure modeling and causal information extraction. We improve the tensor decomposition by a more flexible regularization scheme, and theoretically prove the convergence of the algorithm. Experimental results using synthetic and real EHR datasets (MIMIC-III) demonstrate that the proposed method outperforms state-of-the-art techniques and improves the explanatory power of causal representations.