Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning

Created by
  • Haebom

Author

Wei Liu, Siya Qi, Xinyu Wang, Chen Qian, Yali Du, Yulan He

Outline

This paper proposes NOVER (No-Verifier Reinforcement Learning), a novel framework for reinforcement learning without external verifiers. Conventional incentive learning approaches rely on external verifiers, limiting their applicability in domains like mathematics and coding where verifiers are not readily available. However, NOVER enables incentive learning using only standard supervised learning fine-tuning data. Applicable to a variety of text-to-text tasks, NOVER outperforms similarly sized models distilled from large-scale inference models like DeepSeek R1 671B by 7.7%. Furthermore, it presents new possibilities for large-scale language model optimization, such as inverse incentive learning.

Takeaways, Limitations

•
Takeaways:
◦
We present a novel method to improve the inference ability of language models through reinforcement learning without the need for external verifiers.
◦
Provides a general framework applicable to a variety of text-to-text operations.
◦
Achieve improved performance over existing large-scale inference models.
◦
Suggesting the possibility of new large-scale language model optimization techniques, such as inverse incentive learning.
•
Limitations:
◦
The performance improvements of the proposed method may be limited to specific datasets or tasks.
◦
Further research is needed on NOVER's generalization performance and applicability to various tasks.
◦
Further analysis is needed on the effectiveness of new optimization techniques, such as inverse incentive learning.
👍
Made with Slashpage