Reinforcement learning is very sensitive to hyperparameters, which leads to instability and inefficiency. To solve this problem, hyperparameter optimization (HPO) algorithms have been developed. Population-Based Training (PBT) is an algorithm that has attracted attention for its ability to generate hyperparameter schedules instead of fixed settings. PBT trains multiple agents with different hyperparameters and repeats the process of replacing low-performing agents with variants of superior agents. However, due to this intermediate selection process, PBT focuses on short-term improvements and falls into local optima, which may result in lower performance than general random search in the long term. This paper studies how this greedy problem is related to the evolution frequency (the speed at which selection is made), and proposes MF-PBT (Multiple-Frequencies Population-Based Training), a new HPO algorithm that solves the greedy problem by using subpopulations that evolve at different frequencies. MF-PBT introduces a migration process that transfers information between subpopulations to balance short-term and long-term optimization. Extensive experiments on the Brax suite show that MF-PBT improves sample efficiency and long-term performance without tuning hyperparameters.
