This paper presents a novel framework, MachineLearningLM, that enhances the in-context learning (ICL) capabilities of large-scale language models (LLMs). MachineLearningLM is pretrained using a variety of machine learning (ML) tasks generated from millions of structured causal models (SCMs). Specifically, we infuse the LLM with a decision-making strategy based on random forests to enhance the robustness of numerical modeling. Furthermore, we utilize token-efficient prompts to process 3-6x more examples per context window and achieve up to 50x throughput improvements through batch inference. Consequently, MachineLearningLM based on Qwen-2.5-7B-Instruct outperforms existing powerful LLM baseline models (e.g., GPT-5-mini) by an average of 15% on out-of-distribution tabular data classification tasks across various domains (e.g., finance, physics, biology, and medicine), demonstrating a monotonic increase in accuracy as the number of in-context examples increases (many-shot scaling law). Additionally, we achieved 75.4% performance in MMLU, demonstrating that it retains general conversational skills (knowledge and reasoning).
