Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

$\Texttt{Droid}$: A Resource Suite for AI-Generated Code Detection

Created by
  • Haebom

Author

Daniil Orel, Indraneil Paul, Iryna Gurevych, Preslav Nakov

Outline

In this paper, we present DroidCollection , the most comprehensive open dataset for training and evaluating machine-generated code detectors . DroidCollection contains over a million code samples, seven programming languages, 43 coding model outputs, and at least three real-world coding domains. In addition to entirely AI-generated samples, it also includes code co-written by humans and AI, as well as adversarial samples explicitly crafted to evade detection. We then develop DroidDetect , a set of encoder-specific detectors trained on multi-task objectives using DroidCollection . Experimental results demonstrate that the performance of existing detectors fails to generalize beyond the narrow training data set to diverse coding domains and programming languages. Furthermore, while most detectors can be easily compromised by humanizing the output distribution using superficial prompting and alignment approaches, we demonstrate that training with a small amount of adversarial data can readily address this issue. Finally, we demonstrate that metric learning and uncertainty-based resampling are effective methods for improving detector training in potentially noisy distributions.

Takeaways, Limitations

Takeaways:
Providing a large-scale open dataset (DroidCollection) for training and evaluating machine-generated code detectors.
Addressing the issue of poor generalization performance of existing detectors across various coding domains and programming languages.
Suggesting the possibility of improving detector performance by utilizing adversarial examples.
Demonstrating the effectiveness of metric learning and uncertainty-based resampling techniques.
Limitations:
Despite the diversity of DroidCollection, it may not fully cover all real-world coding domains and programming languages.
The proposed adversarial example generation method may not cover all types of evasion techniques.
The effectiveness of metric learning and uncertainty-based resampling techniques may vary depending on the data distribution.
👍