Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

MMGeoLM: Hard Negative Contrastive Learning for Fine-Grained Geometric Understanding in Large Multimodal Models

Created by
  • Haebom

Author

Kai Sun, Yushi Bai, Zhen Yang, Jiajie Zhang, Ji Qi, Lei Hou, Juanzi Li

MMGeoLM: A Large-Scale Multimodal Model Based on Hard Negative Contrast Learning for Geometric Inference

Outline

To address the inability of ViT-based large multimodal models (LMMs) to capture subtle visual differences in geometric scenarios, this paper proposes a hard-negative contrast learning framework. This framework combines image-based contrast learning using generative hard negatives generated by modifying diagram generation code, and text-based contrast learning using rule-based negatives derived from retrieval-based negatives selected based on modified geometric descriptions and caption similarities. Using the proposed hard-negative training method, Multimodal Math CLIP (MMCLIP), the authors train a visual encoder (CLIP), which in turn trains an LMM for solving geometric problems. Experimental results show that the 7-byte MMGeoLM model significantly outperforms other open-source models on three geometric inference benchmarks, achieving performance comparable to powerful closed-form models such as GPT-4o. Additionally, through analysis of hard negative types, the efficiency of image-based negatives, and training configurations, we gain insights into optimizing the visual encoder training pipeline for fine-grained geometric inference tasks.

Takeaways, Limitations

•
Takeaways:
◦
Improving LMM model performance for solving geometric reasoning problems: MMGeoLM outperforms existing open-source models and achieves performance comparable to GPT-4o.
◦
Demonstrating the effectiveness of a hard negative contrast learning framework: We demonstrate that training a visual encoder using hard negatives contributes to improving geometric reasoning capabilities.
◦
Providing Insights into Training Pipeline Optimization: We propose a method to optimize the visual encoder training pipeline by analyzing factors such as hard negative type, efficiency of image-based negatives, and training configuration.
•
Limitations:
◦
The model size is limited: it was trained with a focus on the 7B model, so its performance on larger models requires further study.
◦
Only performance on specific geometric benchmarks is validated: generalization performance on a wider range of geometric problems requires further evaluation.
◦
Comparisons with closed models have limitations: Comparisons with closed models such as GPT-4o require caution in interpretation due to the lack of information about the model's internal structure and training data.
👍
Made with Slashpage