This paper addresses the problem of existing continuous emotion recognition approaches ignoring emotional ambiguity or treating it as an independent, static variable over time. We propose an ambiguity- aware ordinal emotion representation, a novel framework that captures both the ambiguity and temporal dynamics of emotional expressions. Specifically, we propose an approach that models emotional ambiguity through rates of change and evaluate it using constrained (arousal, preference) and unconstrained (immersion) continuous tracking data from two emotion corpora: RECOLA and GameVibe. Experimental results show that ordinal representations outperform existing ambiguity-aware models on unconstrained labels, achieving the highest Concordance Correlation Coefficient (CCC) and Signed Differential Agreement (SDA) scores. On constrained tracking data, they outperform SDA, demonstrating superior performance in capturing relative changes in annotated emotion traces.
