In this paper, we propose SynC, a synthetic dataset cleaning framework for zero-shot image captioning (ZIC). Existing ZICs utilize synthetic datasets generated by text-to-image (T2I) models to reduce expensive manual annotation work, but the images generated by T2I models often have semantic inconsistencies with their captions. Existing data cleaning techniques focus on removing noisy texts from web crawled data, which is not suitable for the features of synthetic data (well-formed captions, inaccurate images). SynC reassigns captions to images that are semantically most consistent with the captions from the existing image pool. It first retrieves multiple candidate images for each caption, and then selects the optimal image by checking whether the original caption can be retrieved through image-to-text retrieval using alignment scores based on circular consistency. Experimental results show that SynC outperforms various ZIC models and benchmarks (MS-COCO, Flickr30k, NoCaps) and achieves state-of-the-art results.
