Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

RewardAnything: Generalizable Principle-Following Reward Models

Created by
  • Haebom

Author

Zhuohao Yu, Jiali Zeng, Weizheng Gu, Yidong Wang, Jindong Wang, Fandong Meng, Jie Zhou, Yue Zhang, Shikun Zhang, Wei Ye

Outline

This paper points out the Limitations of the reward model (RM), which is essential for large-scale language model (LLM) optimization, and presents a novel approach to overcome it. Existing RMs have the problem that they are trained with fixed preference datasets and cannot adapt to various real-world needs. In this paper, we propose a generalizable RM that dynamically understands and follows reward principles expressed in natural language. To this end, we develop a new benchmark RABench to evaluate generalization ability for various principles, and present RewardAnything, a new RM that is designed and trained to explicitly follow natural language principles. RewardAnything achieves the best performance on existing RM benchmarks, and also shows excellent adaptability to new principles on RABench. In addition, RewardAnything can be seamlessly integrated with existing RLHF methods, and we demonstrate through a case study how to automatically and efficiently align LLMs using only natural language principles.

Takeaways, Limitations

•
Takeaways:
◦
Presenting a generalizable reward model (RewardAnything) that follows reward principles expressed in natural language
◦
Revealing the problem of lack of generalization ability of existing RM and developing a new benchmark RABench
◦
A new method to efficiently sort LLMs based on natural language principles alone
◦
Confirmation of the possibility of seamless integration with existing RLHF methods
◦
Achieving top performance on existing RM benchmarks by simply stating well-defined principles
•
Limitations:
◦
Additional validation of RABench's versatility and representativeness is needed
◦
Additional experiments are needed to determine whether RewardAnything performs well in all situations.
◦
Further research is needed on how to deal with ambiguity or conflicts in natural language principles.
👍
Made with Slashpage