Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Identify, Isolate, and Purge: Mitigating Hallucinations in LVLMs via Self-Evolving Distillation

Created by
  • Haebom

Author

Wenhao Li, Xiu Su, Jingyi Wu, Feng Yang, Yang Liu, Yi Chen, Shan You, Chang Xu

Outline

This paper proposes the Self-Evolving Distillation (SEED) technique to address the hallucination problem in large-scale vision-language models (LVLMs). SEED identifies and removes hallucinations from the internal knowledge of LVLMs, then distills the refined knowledge back into the model, allowing the model to evolve on its own. To address the gap problem of existing distillation methods, a mode-search evolutionary approach is used to capture the dominant modes of the refined knowledge distribution, and an hallucination-removal adapter is used to correct incorrect knowledge in the original model. Experimental results on the LLaVA-1.5 and InternVL2 models demonstrate that SEED is effective in alleviating the hallucination problem, and in particular, it improves the F1 score of LLaVA-1.5 from 81.3 to 88.3 based on the POPE-Random metric.

Takeaways, Limitations

Takeaways:
SEED, a novel method to effectively alleviate the hallucination problem of LVLMs, is proposed.
Solving the problem of long inference time, which is a drawback of existing methods.
Experimentally verified performance improvements on representative LVLM models such as LLaVA-1.5 and InternVL2.
A mode-exploration evolution method and a hallucination removal adapter are presented to complement the existing distillation method Limitations.
Limitations:
The effectiveness of SEED may be limited to specific LVLM models and benchmarks.
Generalization performance to other types of hallucinations or more complex environments requires further study.
Further research may be needed on parameter settings and optimization of the mode search evolution method.
👍