This paper proposes a method for transferring feature representations from a larger teacher model to a lightweight student model. To this end, we mathematically define a novel concept called "perceptual consistency" and propose a loss function that considers differences between data points through rankings. By minimizing this loss function, the student model learns to mimic the way the teacher model "perceives" the input. Specifically, because the student model's representational power is weaker than that of the teacher model, we maintain overall consistency through rankings of differences rather than preserving absolute geometric structure. This "perceptual consistency" extends the rankings defined for finite sets into a probabilistic form, relies on the input distribution, and applies to common dissimilarity metrics. The proposed method outperforms or even surpasses robust baseline methods for representation transfer.
