While CLIP, which aligns image and text embeddings via contrastive learning, offers excellent zero-shot performance, its performance deteriorates in few-shot classification due to inconsistencies within homogeneous modalities. This issue arises from the disparity between modalities and CLIP's cross-modality training objective, and its inability to calibrate the embedding space makes direct comparisons between images difficult. SeMoBridge addresses this issue by mapping images to text modalities, a lightweight and robust approach that preserves semantic content. SeMoBridge is a closed-form, combined image and text alignment loss that can be optionally trained with multimodality supervision. SeMoBridge-T outperforms other methods with a shorter training time, particularly in low-data scenarios.
