Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning

Created by
  • Haebom

Author

Xiaoyun Zhang, Xiaojian Yuan, Di Huang, Wang You, Chen Hu, Jingqing Ruan, Kejiang Chen, Xing Hu

Outline

This paper addresses the policy entropy decay problem that occurs during Reinforcement Learning with Verifiable Rewards (RLVR) training to improve the inference performance of large-scale language models (LLMs). Policy entropy decay occurs when policies become overly deterministic, hindering exploration and limiting inference performance. This study proposes a framework called Adaptive Entropy Regularization (AER), which dynamically balances exploration and exploitation through difficulty-aware coefficient allocation, an initially anchored target entropy, and dynamic global coefficient adjustment. Experimental results on several mathematical inference benchmarks demonstrate that AER outperforms existing methods, improving both inference accuracy and exploration.

Takeaways, Limitations

•
Takeaways:
◦
A novel methodology for improving the inference ability of RLVR-based LLM (AER).
◦
We reexamine the effectiveness of entropy regularization and demonstrate the potential for stable performance improvements through dynamic adjustment.
◦
Achieving a more effective exploration-exploitation balance through difficulty awareness, initially anchored target entropy, and dynamic coefficient tuning.
◦
Demonstrated superior performance compared to existing methodologies in various mathematical inference benchmarks.
•
Limitations:
◦
It is difficult to conclude general performance based solely on experiments on specific mathematical inference benchmarks.
◦
Further analysis of the detailed components of AER (how difficulty is measured, setting target entropy, etc.) is needed.
◦
Further research is needed on the possibility of extending this to other LLM-based tasks (e.g., natural language processing).
👍
Made with Slashpage