Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learning

Created by
  • Haebom

Author

Teeradaj Racharak, Chaiyong Ragkhitwetsagul, Chommakorn Sontesadisai, Thanwadee Sunetnanta

Outline

This paper presents MMT4NL, a software testing-based framework for evaluating the reliability of in-context learning (ICL) of large-scale language models (LLMs). MMT4NL exploits adversarial examples and software testing techniques to identify vulnerabilities in ICLs. It treats LLMs as software and generates modified adversarial examples from a test set to quantify and identify bugs in ICL prompts. Experiments on sentiment analysis and question-answering tasks reveal various linguistic bugs in state-of-the-art LLMs.

Takeaways, Limitations

Takeaways:
A new framework (MMT4NL) for evaluating the ICL reliability of LLM using software testing techniques is presented.
We demonstrate that adversarial example generation can effectively identify vulnerabilities in LLM.
Contributed to improving the performance of LLM by uncovering various linguistic bugs.
Limitations:
MMT4NL's applicability is limited to sentiment analysis and question-answering tasks. Generalizability to other tasks needs to be verified.
Further research is needed to determine the efficiency and scalability of the proposed framework.
Further validation is needed to ensure that all types of linguistic bugs can be caught.
👍