This paper explores leveraging large-scale language models (LLMs) to annotate document usefulness and reduce reliance on expensive manual annotations in training retrieval and augmented retrieval generation (RAG) systems. To bridge the gap between retrieval relevance and generative usefulness, we use LLMs to annotate document usefulness. To effectively utilize multiple positive samples per query, we propose a novel loss function that maximizes their aggregated marginal likelihood. We use the Qwen-2.5-32B model to annotate the MS MARCO dataset for usefulness and conduct retrieval experiments on MS MARCO and BEIR, as well as RAG experiments on MS MARCO QA, NQ, and HotpotQA. Our experimental results show that LLM-generated annotations improve out-of-domain retrieval performance and RAG results compared to models trained solely on manual annotations or subsets of QA metrics. Furthermore, we achieve performance comparable to that achieved with fully manual annotations by combining LLM annotations with 20% of the manual annotations. This study presents a comprehensive approach for leveraging LLM annotations to initialize QA systems on new corpora.
