Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on Math Textbook

Created by
  • Haebom

Author

Eason Chen, Chuangji Li, Shizhuo Li, Zimo Xiao, Jionghao Lin, Kenneth R. Koedinger

Outline

This study investigates a technology-enhanced learning environment that facilitates the retrieval of relevant learning content for questions during self-directed learning, specifically exploring information retrieval methods utilizing large-scale language models (LLMs). Targeting undergraduate mathematics textbooks, we compare and analyze Augmented Search Generation (RAG) and GraphRAG, which utilizes knowledge graphs, for page-level question answering. Using a dataset of 477 question-answer pairs, we evaluate the retrieval accuracy and generated answer quality (F1 score) of RAG and GraphRAG. The results show that embedding-based RAG outperforms GraphRAG. Furthermore, attempts at re-ranking using LLMs resulted in performance degradation and hallucinations. This study highlights the potential and challenges of page-level retrieval systems in educational settings and highlights the need for sophisticated retrieval methods for building AI tutoring solutions.

Takeaways, Limitations

Takeaways:
Embedding-based RAG outperforms GraphRAG in page-level question-answering of mathematics textbooks.
To verify the feasibility of page-level information retrieval systems in educational environments.
Presenting the possibility of developing AI tutoring solutions.
Limitations:
GraphRAG tends to over-retrieve irrelevant content.
Performance degradation or hallucinations occur during the re-ranking process using LLM.
Raises the need for development of more sophisticated search methods.
👍