In this paper, we propose a retrieval-based question answering (QA) pipeline to address the problems of hallucination (generated false information) and high cost of proprietary models that limit the utilization of large-scale language models (LLMs) in customer support areas, and explore the balance between human intervention and automation. Using a dataset of questions about Samsung smart TV user manuals, we show that synthetic data generated by LLMs are more effective in reducing hallucination of fine-tuned models than crowdsourced data. We also compare self-learning (fine-tuning with the output of the model) with knowledge distillation (fine-tuning with the output of a more powerful model, e.g., GPT-4), and confirm that self-learning achieves similar hallucination reduction effects, which we speculate is due to increased exposure bias in the case of knowledge distillation, which we support with additional analysis. In addition, we improve the robustness against unanswerable questions and retrieval failures by providing context-sensitive “I don’t know” responses. These results demonstrate that synthetic data and self-learning using open-source models can be used to build scalable and cost-effective QA systems, reducing the reliance on proprietary tools or expensive human annotations.
