This paper proposes HARDY-MER, a hardness-aware dynamic curriculum learning framework, to address the multimodal emotion recognition (MER) problem with missing modalities. Existing missing modality reconstruction methods have limitations in that they fail to account for differences in reconstruction difficulty across samples. HARDY-MER consists of two stages: assessing sample difficulty and strategically emphasizing difficult samples for learning. Sample difficulty is measured through a multi-perspective difficulty assessment mechanism that considers direct difficulty (modality reconstruction error) and indirect difficulty (cross-modal mutual information). The learning process is regulated through a retrieval-based dynamic curriculum learning strategy that searches for samples with similar semantic information and adjusts the learning weight between easy and difficult samples. Experimental results on benchmark datasets demonstrate that HARDY-MER outperforms existing methods.
