This is a page that curates AI-related papers published worldwide. All content here is summarized using Google Gemini and operated on a non-profit basis. Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.
Detecting Machine-Generated Texts: Not Just “AI vs Humans” and Explainability is Complicated
Created by
Haebom
Author
Jiazhou Ji, Ruizhe Li, Shujun Li, Jie Guo, Weidong Qiu, Zheng Huang, Chiyu Chen, Xiaoyu Jiang, Xinru Lu
Outline
This paper addresses the limitations of the conventional binary classification method (human vs. AI) for detecting texts generated by large-scale language models (LLMs) and proposes a ternary classification method that adds an 'undecidable' category to improve the explainability of the detection results of LLM-generated texts. The study creates four new datasets consisting of various LLMs and human-written texts and evaluates the performance of state-of-the-art (SOTA) detection methods to identify which LLMs generate difficult-to-detect texts. In addition, we evaluate the performance of state-of-the-art detectors using a dataset in which three human annotators assigned ternary classification labels along with their explanations, and analyze the explainability to provide guidelines for the development of detection systems with improved explainability.
Takeaways, Limitations
•
Takeaways:
◦
We propose a ternary classification method that overcomes the limitations of the existing binary classification method and increases explainability.
◦
State-of-the-art LLM detection methods and challenging text-generated LLMs are tested on new datasets.
◦
Providing guidelines for developing improved detection systems through explainability analysis of detection results.
◦
LLM emphasizes the importance of explainability in the field of generative text detection.
•
Limitations:
◦
Further research is needed to determine the generalizability of the proposed ternary classification method.
◦
Limitations on the size and diversity of the dataset used.
◦
Consideration should be given to the possible influence of human annotator subjectivity on the results.
◦
Further validation of the practical applicability and effectiveness of the proposed guidelines is needed.
