Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

EZ-Sort: Efficient Pairwise Comparison via Zero-Shot CLIP-Based Pre-Ordering and Human-in-the-Loop Sorting

Created by
  • Haebom

Author

Yujin Park, Haejun Chung, Ikbeom Jang

Outline

This paper explores the tendency to favor pairwise comparisons over absolute ranks or sequence classifications for improved reliability in subjective or challenging annotation tasks. While traditional pairwise comparisons require a large number of annotations (O(n^2)), recent research has significantly reduced the annotation burden (O(n log n)) by actively sampling pairwise comparisons using alignment algorithms. In this paper, we further improve annotation efficiency by (1) roughly pre-aligning items hierarchically using the Contrastive Language-Image Pre-training (CLIP) model without training, and (2) replacing easy and obvious human comparisons with automated comparisons. The proposed EZ-Sort first generates a CLIP-based zero-shot pre-alignment, then initializes bucket-aware Elo scores, and finally runs uncertainty-based human-involved MergeSort. We validated our approach using various datasets, including Face Age Estimation (FGNET), Historical Image Chronology (DHCI), and EyePACS (EyePACS). As a result, EZ-Sort maintained or improved inter-rater reliability while reducing human annotation costs by 90.5% compared to full pairwise comparison and by 19.8% (when n = 100) compared to existing studies. These results demonstrate that combining CLIP-based prior information and uncertainty-aware sampling yields an efficient and scalable pairwise ranking solution.

Takeaways, Limitations

•
Takeaways:
◦
We significantly improved the efficiency of pairwise comparison tasks by leveraging zero-shot pre-alignment and uncertainty-based sampling with CLIP.
◦
We achieved a 90.5% reduction in annotation costs compared to full pairwise comparisons and a 19.8% reduction compared to existing studies.
◦
Improved efficiency while maintaining or improving inter-rater reliability.
◦
We verified the generalization performance of the proposed method through validation on various datasets.
•
Limitations:
◦
It depends on the performance of the CLIP model, and the limitations of the CLIP model may affect the performance of EZ-Sort.
◦
Because the accuracy of automated comparisons is not always guaranteed, errors may occur.
◦
It may be optimized for a specific type of dataset and may perform poorly on other types of datasets.
◦
Further research may be needed on scalability to large datasets.
👍
Made with Slashpage