Chatbot Arena is a platform for evaluating large-scale language models (LLMs) by having users vote for their preferred response between two anonymous models. This paper demonstrates that crowdsourced voting can be manipulated to artificially boost or lower the ranking of a specific model. First, we introduce a simple manipulation strategy that focuses only on voting for a specific model and point out its inefficiency. To overcome this, we propose a comprehensive manipulation strategy that leverages Chatbot Arena's Elo rating mechanism to manipulate votes in matches not directly related to a specific model, thereby influencing its ranking. Experiments using 1.7 million existing vote data demonstrate that this comprehensive manipulation strategy can improve model rankings with just a few hundred new votes. While we evaluate several defense mechanisms, we emphasize the importance of preventing vote manipulation.