In this paper, we present a novel reinforcement learning algorithm called sequence reinforcement learning (SRL). SRL is designed to generate a sequence of actions for a given input state, enabling effective control even at low decision frequencies. We address the difficulty of learning action sequences by using models and action-critic architectures at different time scales. In particular, we propose a “temporal replay” mechanism in which the critic uses the model to estimate intermediate states between basic actions, providing learning signals for each action in the sequence. After learning is completed, the action generator generates action sequences independently of the model, achieving model-free control at low frequencies. To better evaluate the performance at various decision frequencies, we introduce the frequency-average score (FAS) metric, and demonstrate the superiority of SRL over existing algorithms in continuous control tasks.
