[공지사항]을 빙자한 안부와 근황 
Show more

Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection

Created by
  • Haebom

Author

Aleksandr Gashkov, Aleksandr Perevalov, Maria Eltsova, Andreas Both

Outline

This paper aims to improve the quality of the process of converting natural language questions into SPARQL queries (Query Building) in a knowledge graph question answering (KGQA) system using a large-scale language model (LLM). Existing LLM-based KGQA systems have a limitation in that they cannot know whether the training data of LLM includes benchmarks or knowledge graphs. Therefore, in this paper, we present a new methodology to evaluate the quality of SPARQL query generation of LLM under various conditions, such as (1) zero-shot SPARQL generation, (2) knowledge injection, and (3) anonymized knowledge injection. Through this, we estimate for the first time the impact of LLM training data on improving QA quality, and evaluate the generalizability of the method by distinguishing between the actual performance of LLM and the effect of training data memorization. The proposed method is portable and robust, and can be applied to various knowledge graphs, providing consistent insights.

Takeaways, Limitations

•
Takeaways:
◦
A new methodology is presented to quantitatively analyze the impact of LLM learning data on KGQA performance.
◦
Establishing criteria to distinguish between the actual performance of LLM and the mere memorization effect
◦
Provides a highly portable evaluation method applicable to various knowledge graphs and LLMs
◦
Contributes to improving the reliability and generalization performance of the KGQA system
•
Limitations:
◦
There are not enough experimental results to evaluate the performance of the proposed method (this summary alone does not allow for a judgment).
◦
There may be bias towards certain knowledge graphs or LLMs (not possible to determine from this summary alone)
◦
Lack of detailed description of the specific methods and effects of “anonymous knowledge injection” (it is impossible to judge from this summary alone)
👍
Made with Slashpage