SPICE is a scalable, automated pipeline for generating high-quality labeled datasets essential for learning and evaluating foundational models in software engineering. It automatically annotates SWE-bench-style datasets with problem clarity, test coverage, and effort estimation. It combines context-aware code exploration, evidence-based prompting, and multi-pass consensus to produce labels that closely resemble expert annotations. It is built on the experience of labeling over 800 SWE-Gym instances and achieves high agreement with human-labeled SWE-bench Verified data. It dramatically reduces the cost of labeling 1,000 instances from approximately $100,000 for manual annotation to $5.10. We also release SPICE Bench, a new dataset consisting of 6,802 SPICE-labeled instances from 291 open-source projects in SWE-Gym.
