This paper identifies three major drawbacks of reinforcement learning (RL) that rely solely on numerical rewards: performance plateaus, limited effectiveness of self-reflection, and persistent failures. To overcome these drawbacks, we propose Critique-GRPO, a novel reinforcement learning framework that integrates natural language critique. Critique-GRPO performs policy optimization by simultaneously leveraging both numerical and natural language feedback, and employs a shaping function that reinforces learning for correct corrections and penalizes incorrect ones. Experimental results using the Qwen2.5 and Qwen3 models show that Critique-GRPO consistently outperforms existing supervised learning and RL-based fine-tuning methods on eight challenging mathematics, STEM, and general reasoning tasks, improving the pass@1 scores by approximately 4.4% (Qwen2.5-7B-Base) and 3.8% (Qwen3-8B), respectively, on average. In particular, the self-improvement effect through self-criticism was excellent, achieving a pass@1 improvement of +16.7% compared to GRPO (AIME 2024).
