Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

DISCO: Diversifying Sample Condensation for Efficient Model Evaluation

Created by
  • Haebom

Author

Alexander Rubinstein, Benjamin Raible, Martin Gubri, Seong Joon Oh

Outline

Evaluation of modern machine learning models is expensive. Diversifying Sample Condensation (DISCO) is a novel method that reduces evaluation costs by selecting samples that maximize the diversity of model responses. It selects samples based on model mismatch. It is conceptually simpler than existing methods and delivers the best prediction results on the MMLU, Hellaswag, Winogrande, and ARC benchmarks.

Takeaways, Limitations

Takeaways:
Increasing accessibility to machine learning research by reducing evaluation costs.
Accelerate the pace of innovation and reduce environmental impact.
Achieving superior performance in a conceptually simple way.
Theoretically, it is proven that the discrepancy between models is the optimal selection rule.
Achieved SOTA on MMLU, Hellaswag, Winogrande, and ARC benchmarks.
Limitations:
Specific Limitations for anchor sample selection is not specified in the paper.
Further research is needed to determine the generalizability of DISCO.
Detailed information on specific experimental results and comparative analyses is required.
👍