Advances in reinforcement learning (RL) have improved the inference capabilities of large-scale language models (LLMs). However, Group Relative Policy Optimization (GRPO), a lightweight variant of Proximal Policy Optimization (PPO), has limited its effectiveness for complex inference tasks due to limitations in exploration and training instability. To address these issues, we propose the EFRame framework, which incorporates exploration, filtering, and replay. EFRame enables deeper and more targeted exploration with additional rollouts, removes low-quality samples to stabilize gradients and accelerate training, and amplifies rare but informative trajectories through experience replay to achieve stable convergence. EFRame demonstrates a 37.9% relative performance improvement over GRPO on the Geometry3K geometry problem-solving benchmark. EFRame supports fine-grained sample classification and precise entropy control, highlighting its potential as a powerful solution for advancing deeper LLM inference.
