Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Improving Your Model Ranking on Chatbot Arena by Vote Rigging

Created by
  • Haebom

Author

Rui Min, Tianyu Pang, Chao Du, Qian Liu, Minhao Cheng, Min Lin

Outline

Chatbot Arena is a platform for evaluating large-scale language models (LLMs) by having users vote for their preferred response between two anonymous models. This paper demonstrates that crowdsourced voting can be manipulated to artificially boost or lower the ranking of a specific model. First, we introduce a simple manipulation strategy that focuses only on voting for a specific model and point out its inefficiency. To overcome this, we propose a comprehensive manipulation strategy that leverages Chatbot Arena's Elo rating mechanism to manipulate votes in matches not directly related to a specific model, thereby influencing its ranking. Experiments using 1.7 million existing vote data demonstrate that this comprehensive manipulation strategy can improve model rankings with just a few hundred new votes. While we evaluate several defense mechanisms, we emphasize the importance of preventing vote manipulation.

Takeaways, Limitations

Takeaways: Demonstrates the vulnerability of crowdsourcing-based LLM assessment platforms, such as Chatbot Arena. Demonstrates that even a relatively small amount of vote manipulation can significantly alter model rankings through a comprehensive manipulation strategy. Emphasizes the importance of developing anti-voting technology to ensure the reliability of LLM assessment platforms.
Limitations: Lack of detailed analysis of the effectiveness of the proposed defense mechanism. Lack of comprehensive analysis of various types of manipulation strategies. This study is based on analysis of existing data, not direct manipulation attempts on the actual Chatbot Arena system. Lack of in-depth analysis of factors affecting the success rate of specific manipulation strategies (e.g., voter participation, model characteristics, etc.).
👍