In this paper, we propose a novel framework to address the hallucination problem of large-scale language models (LLMs) that answer questions using contexts retrieved from vectorized databases. Hallucinations in the responses of existing LLMs are a major problem that reduces the reliability and are difficult to detect, especially when there is no correct answer. To address this issue, we present a framework that integrates multiple small-scale language models to verify the responses generated by LLMs. We detect hallucinations by dividing the responses into sentences and analyzing the probability of "Yes" tokens generated by multiple models for each sentence. Experimental results using a dataset of more than 100 questions, answers, and contexts show that the proposed framework improves the F1 score of hallucination detection by 10%, demonstrating that it is a scalable and efficient solution applicable to both academia and practice.
