To address the challenges of building high-quality datasets for specialized tasks, this paper proposes Corpus Retrieval and Augmentation for Fine-Tuning (CRAFT), a method that generates synthetic datasets based on a small number of user-generated shots. CRAFT uses a large-scale public web crawl corpus and similarity-based document retrieval to find relevant documents, and leverages a directive-tuned giant language model (LLM) to augment the retrieved documents with user-defined task samples. Experiments on four diverse tasks—biology, medicine, common-sense question answering (QA), and summarization—demonstrate that CRAFT efficiently generates large, task-specific training datasets, outperforming or equaling a standard LLM on the QA task and achieving a 46-point preference improvement over models trained on existing human-curated data on the summarization task. Furthermore, it outperforms other synthetic dataset generation methods, such as Self-Instruct and Evol-Instruct, and maintains robust performance even when the quality of the initial few shots varies.
