Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks

Created by
  • Haebom

Author

Rahul Ramachandran, Ali Garjani, Roman Bachmann, Andrei Atanov, O\u{g}uzhan Fatih Kar, Amir Zamir

Outline

In this paper, we benchmarked the performance of popular multimodal-based models such as GPT-4o, o4-mini, Gemini 1.5 Pro, Gemini 2.0 Flash, Claude 3.5 Sonnet, Qwen2-VL, and Llama 3.2 on computer vision tasks (semantic segmentation, object detection, image classification, and depth and surface normal prediction) using standard datasets such as COCO and ImageNet. Considering that the models are optimized for text output and many of them only allow API access, we built a standardized benchmarking framework that converts standard vision tasks into text-prompt-based tasks through prompt chaining. As a result, although it falls short of existing expert models, it shows considerable performance on various tasks, and in particular, it outperforms geometric tasks on semantic tasks. GPT-4o shows the best performance among non-inference models, and inference models show improved performance on geometric tasks, but the latest GPT-4o with image generation function shows problems such as hallucination and spatial misalignment.

Takeaways, Limitations

•
Takeaways:
◦
Multimodal-based models demonstrate significant generalization ability in various computer vision tasks despite being image-text-based learning.
◦
We demonstrate that performance on semantic tasks is superior to that on geometric tasks.
◦
Suggesting potential performance improvements through the use of prompt engineering and inference models.
◦
Validating the potential of modern multimodal models including GPT-4o.
•
Limitations:
◦
The performance of multimodal-based models is significantly lower than that of expert computer vision models.
◦
Highly reliant on prompt chaining techniques and sensitive to prompt changes.
◦
Problems such as hallucinations and spatial misalignment occur in models with image generation capabilities.
◦
Difficulty in fully evaluating the model's performance due to restricted API access.
👍
Made with Slashpage