This paper presents a novel method for learning control policies by combining pre-training using offline data and online fine-tuning using reinforcement learning. To address the problem that useful behaviors of offline policies can be lost in the early stages of traditional online learning, we propose a technique that uses an offline-trained policy as a candidate policy in a policy set and expands the policy set by adding another policy for further learning. The two policies are adaptively configured to interact with the environment, and the offline policy is fully maintained during online learning. This allows the offline policy to naturally engage in exploration while preserving its useful behaviors, while also allowing the newly added policy to learn new useful behaviors. Experimental results on various tasks demonstrate the effectiveness of the proposed method.
