BMMR is a large-scale multilingual, multimodal, multidisciplinary inference dataset containing 110,000 university-level questions for the development and evaluation of large-scale multimodal models (LMMs). It covers 300 subjects defined by UNESCO and consists of questions in various formats such as multiple choice, fill-in-the-blank, and short-answer questions, and data from various sources such as books, exams, and quizzes. It is curated and filtered through a scalable framework based on human intervention, and each instance is associated with a high-quality inference path. The dataset is divided into BMMR-Eval, which consists of 20,458 high-quality instances for comprehensively evaluating the knowledge and inference ability of LMMs, and BMMR-Train, which consists of 88,991 instances to support further research and development. We also propose a process-based multidisciplinary verifier (BMMR-Verifier) for accurate and fine-grained inference path evaluation. Experimental results on 24 models show that even state-of-the-art models have significant room for improvement in BMMR-Eval, inference models outperform LMMs only on specific subjects, and open-source models underperform proprietary models, but fine-tuning with BMMR-Train reduces the performance gap. Further in-depth studies, including analysis of inference chains using BMMR-Verifier, reveal the current challenges LMMs face in multidisciplinary inference. The dataset will be made public.
