Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering

Created by
  • Haebom

Author

Samuel Abedu, SayedHassan Khatoonabadi, Emad Shihab

Outline

To leverage software repository data, which contains valuable information for understanding the software development process, this study integrated a knowledge graph with an LLM-based chatbot to improve the accuracy of repository-related questions. A two-step approach was used: building a knowledge graph from repository data and combining it with the LLM. The results showed that the LLM's inference capabilities led to errors in answering 150 questions of varying difficulty on five open source projects. However, applying few-shot chain-of-thought prompting improved accuracy to 84%. The results outperformed MSRBot and GPT-4o-search-preview, demonstrating its efficiency and usability in user studies.

Takeaways, Limitations

Takeaways:
We demonstrate that combining LLM and knowledge graphs is an effective way to improve repository data accessibility.
Improving LLM's reasoning skills through few-shot chain-of-thought prompting.
The proposed method outperforms existing methods (MSRBot, GPT-4o-search-preview).
The usefulness and effectiveness of the proposed method were verified through user research.
Limitations:
Initial results show that errors occur depending on the inference ability of LLM.
(Additional information about specific Limitations is not specified in the abstract)
👍