This paper highlights the Limitations flaw in SWE-Bench Verified, a benchmark for evaluating software engineering capabilities of large-scale language models (LLMs). While recent LLMs demonstrate high performance on SWE-Bench, this may be due to data memorization or contamination rather than actual problem-solving ability. To verify this, the paper presents two diagnostic tasks: identifying file paths based solely on issue descriptions and reproducing functions based solely on the current file context and issue descriptions. Experimental results show that while state-of-the-art models exhibit high accuracy on data included in SWE-Bench, their accuracy drops sharply on data not included, raising concerns about the reliability of SWE-Bench's evaluation results. This highlights the need for a more robust and contamination-resistant benchmark for evaluating LLM coding capabilities.
