Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Created by
  • Haebom

Author

Joel Becker, Nate Rush, Elizabeth Barnes, David Rein

Outline

This paper conducted a randomized controlled trial (RCT) between February and June 2025 to understand the impact of AI tools on the productivity of experienced open source developers. Sixteen developers (with a medium level of AI experience) performed 246 tasks on projects with an average of 5 years of experience. Each task was randomly assigned whether or not to allow the use of AI tools, and when allowed, they primarily used Cursor Pro and Claude 3.5/3.7 Sonnet. Developers expected that allowing the use of AI tools would reduce task completion time by 24%, but in reality, it increased it by 19%. This result is also contrary to the predictions of economists (who expected a 39% reduction) and ML experts (who expected a 38% reduction). The study analyzed 20 characteristics that could affect the slowdown effect (e.g., project size, quality criteria, and developers’ experience with AI tools) and concluded that although the influence of experimental artifacts cannot be completely ruled out, the robustness of the slowdown effect across the analysis makes it difficult to explain it solely as a problem of the experimental design.

Takeaways, Limitations

Takeaways: We experimentally demonstrated that AI tools could actually reduce the productivity of experienced developers by early 2025. This showed a large gap between the predictions of economics and ML experts and reality. It suggests that a cautious approach is needed to the productivity effects of introducing AI tools.
Limitations: The number of experimental participants is limited (16). The influence of experimental artifacts cannot be completely ruled out. The results may be limited to specific AI tools and development environments. Further research is needed on various AI tools, developer experience levels, and project types.
👍