In this paper, we propose a self-guided process reward optimization (SPRO) framework to address the high computational cost of process reinforcement learning (PRL), which has shown significant potential for improving the inference capability of large-scale language models (LLMs), and the lack of a unified theoretical framework for process-level advantage estimation. SPRO enables process-aware RL through two key innovations: theoretically proving that process rewards can be derived from the policy model itself, and introducing well-defined cumulative process rewards and masked-step advantage (MSA) to enable strict step-by-step action advantage estimation within a shared prompt sampling group. Experimental results show that SPRO achieves 3.4 times higher training efficiency and 17.5% improved test accuracy than the conventional GRPO. In addition, we demonstrate sufficient exploration and prevention of reward hacking by reducing the average response length by about 1/3 while maintaining stable and high policy entropy throughout the training process. In particular, SPRO is advantageous for industrial implementation because it does not incur additional computational costs compared to result-supervised RL methods such as GRPO.
