This paper introduces Time-IMM, a novel dataset that closely reflects real-world environments, and the benchmark library IMM-TSF, to address irregular, multimodal, and missing-value-heavy time series data encountered in real-world applications such as healthcare, climate modeling, and finance. Time-IMM encompasses nine types of time series irregularity categorized by trigger-based, constraint-based, and artifact-based mechanisms, while IMM-TSF enables asynchronous integration and realistic evaluation. IMM-TSF includes a timestamp-text fusion module and a multimodal fusion module, supporting recency-aware averaging and attention-based integration strategies. Experimental results demonstrate that explicit multimodal modeling significantly improves forecasting performance in irregular multimodal time series data.