This is a page that curates AI-related papers published worldwide. All content here is summarized using Google Gemini and operated on a non-profit basis. Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.
This paper points out that existing multimodal large-scale language model (MLLM) research has focused solely on general visual understanding and has overlooked the ability to integrate object-related textual information to perform context-aware multimodal understanding (domain-level context-aware multimodal understanding, RCMU). To address this, we define an RCMU task that requires integrating image content and textual information of a region or object to respond to user commands. We propose a domain-level context-aware visual guidance coordination (RCVIT) method that integrates object information into the model input, enabling the bounding box coordinates to effectively connect the visual and textual content of the object. Furthermore, we introduce the RCMU dataset, a large-scale visual guidance coordination dataset covering various RCMU tasks, and propose RC&P-Bench, a comprehensive benchmark for evaluating the performance of MLLMs on RCMU and multimodal personalized understanding tasks. We also propose reference-free evaluation metrics for comprehensive and granular evaluation of domain-level context-aware image explanations. Finally, we develop the RC-Qwen2-VL model by applying it to the RCVIT and RCMU datasets. Experimental results demonstrate that the model achieves excellent performance on multiple RCMU tasks and demonstrates successful applications in multimodal RAG and personalized conversations. The data, model, and benchmarks are available in https://github.com/hongliang-wei/RC-MLLM .
We present a new challenge called RCMU, which integrates visual and textual information of objects, and propose the RCVIT methodology to solve this problem.
◦
We provided the RCMU dataset, a large-scale dataset for RCMU tasks, and RC&P-Bench, a benchmark for performance evaluation.
◦
We improve the evaluation of domain-level context-aware image descriptions by proposing a reference-free evaluation metric.
◦
The RC-Qwen2-VL model has demonstrated excellent performance in RCMU tasks and multi-modal applications.
•
Limitations:
◦
Further review may be necessary regarding the size and diversity of the RCMU dataset.
◦
Further experiments may be needed to evaluate the generalization performance of the proposed RCVIT methodology.
◦
There may be a lack of discussion on Limitations of evaluation metrics without reference.
◦
Only results for a specific model (Qwen2-VL) are presented, so further research is needed to determine generalizability to other models.
