Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Mining the Mind: What 100M Beliefs Reveal About Frontier LLM Knowledge

Created by
  • Haebom

Author

Shrestha Ghosh, Luca Giordano, Yujia Hu, Tuan-Phong Nguyen, Simon Razniewski

Outline

LLM has revolutionized NLP and AI tasks, and factual knowledge plays a particularly crucial role. This paper deeply analyzes the factual knowledge of state-of-the-art LLM using GPTKB v1.5 (100 million knowledge bases) based on GPT-4.1. We find that its accuracy is lower than previous benchmarks, it differs significantly from existing knowledge bases, and it suffers from major issues such as inconsistency, ambiguity, and illusions.

Takeaways, Limitations

Conducting an in-depth analysis of the factual knowledge of cutting-edge LLMs
Leveraging GPTKB v1.5, the factual knowledge base of GPT-4.1
We found that the model's accuracy was lower than existing benchmarks and differed from existing knowledge bases.
Identify problems such as inconsistencies, ambiguities, and illusions
Presenting research opportunities for future factual LLM knowledge
👍