This paper presents an improved visual-verbal search augmented generation (RAG) approach for solving the knowledge-based visual question answering (KB-VQA) problem. Existing visual-verbal RAG systems struggle to achieve effective multimodal retrieval due to the diverse modalities and granularity of knowledge. To address these challenges, we propose a multimodal RAG system that proceeds in multiple stages, from coarse to fine. First, we perform multimodal retrieval tailored to the knowledge granularity, then we leverage multimodal information to select top-level entities. Finally, we use text reranking to select the most appropriate granular information for generation. We demonstrate that our RAG system contributes to the advancement of KB-VQA systems by achieving state-of-the-art retrieval performance and competitive answering results on the InfoSeek and Encyclopedic-VQA benchmarks.
