Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

RePPL: Recalibrating Perplexity by Uncertainty in Semantic Propagation and Language Generation for Explainable QA Hallucination Detection

Created by
  • Haebom

Author

Yiming Huang, Junyan Zhang, Zihao Wang, Biquan Bie, Yunzhong Qiu, Yi R. Fung, Xinlei He

Outline

This paper proposes RePPL, a novel method for solving the hallucination problem in large-scale language models (LLMs). Existing hallucination detection methods focus on uncertainty measurement, but fail to explain the cause of hallucinations. To overcome this limitation, we calculate a token-level uncertainty score by considering both uncertainty arising during semantic propagation and language generation. These scores are then aggregated as a perplexity-style logarithmic mean to produce an overall hallucination score. Our method demonstrates excellent performance, achieving an average AUC of 0.833 on various QA datasets and state-of-the-art models, demonstrating the utility of token-level uncertainty scores in explaining the cause of hallucinations.

Takeaways, Limitations

•
Takeaways:
◦
A novel approach to solving the LLM's hallucination problem (taking into account uncertainty in the process of meaning propagation and language production)
◦
Token-level uncertainty scores provide a possible explanation for the cause of hallucinations.
◦
Excellent hallucination detection performance (average AUC 0.833) on various QA datasets and state-of-the-art models.
◦
Discovering and exploiting confusing patterns of hallucinations
•
Limitations:
◦
RePPL's performance evaluation is limited to a specific QA dataset and state-of-the-art models. Additional experiments on a wider range of datasets and models are needed.
◦
Analysis of hallucination patterns using token-level uncertainty scores is still in its infancy and requires further in-depth analysis and validation.
◦
Further research is needed on the generalization performance and scalability of the proposed method.
👍
Made with Slashpage