AI coding assistance tools have become essential for software development, and migrating and modernizing codebases to keep pace with the evolving software ecosystem is crucial. This paper introduces FreshBrew, a new benchmark for project-level Java migration, to systematically evaluate the effectiveness of AI agents. We benchmarked several state-of-the-art LLMs across 228 repositories and compared them with existing rule-based tools. Our results show that Gemini 2.5 Flash successfully migrated 52.3% of projects to JDK 17. This provides new insights into the strengths and limitations of current agent-based approaches and establishes a foundation for evaluating reliable code migration systems.