This paper evaluates the performance of the Vision-Language Model (VLM) on a fundamental aspect of natural language semantics: the ability to form new meanings by combining existing parts. We note that VLMs, such as CLIP, tend to represent images in a "bag-of-words" manner, failing to adequately capture compositional meaning. We investigate whether a generative classifier, a diffusion model-based classifier, can overcome this limitation. We evaluate the ability of three models—the Diffusion Classifier, CLIP, and ViLT—to combine objects, attributes, and relationships in zero-shot learning (ZSL) and generalized zero-shot learning (GZSL) environments. Our experimental results demonstrate that while the Diffusion Classifier and ViLT perform well on concept combination tasks, all models struggle with relational GZSL tasks, highlighting the challenges of VLM in relational inference. Analysis of the CLIP embedding suggests that the difficulty stems from the excessive similarity in the representation of relational concepts such as "left" and "right."
