In this paper, we propose a novel framework, Spectral Pairwise Embedding Comparison (SPEC), for comparing different feature embedding models. Unlike previous studies that mainly focus on numerical comparisons based on classification performance, SPEC analyzes the disparity in the clustering of sample groups within the embedding space to analyze the differences between embeddings analytically. It exploits kernel matrices derived from two embeddings and detects sample clusters that are captured differently by the two embeddings through eigendecomposition of the difference kernel matrix. We present a scalable implementation whose computational complexity increases linearly with the sample size, and introduce an optimization problem to align two embeddings so that clusters identified in one embedding are also captured in the other. We present numerical results demonstrating the application of SPEC to compare and align embeddings on large-scale datasets such as ImageNet and MS-COCO.
