This paper presents a novel framework, MachineLearningLM, that enhances the context-sensitive learning (ICL) capabilities of large-scale language models (LLMs). MachineLearningLM is pretrained using a variety of machine learning (ML) tasks generated from millions of structured causal models (SCMs). Specifically, it infuses LLM with a random forest-based decision-making strategy to enhance the robustness of numerical modeling, and enhances throughput by processing more examples per context window through token-efficient prompts. Experimental results demonstrate that MachineLearningLM outperforms existing robust LLM baseline models by an average of 15% on non-distributed tabular data classification tasks across various domains, exhibiting a remarkable multi-shot scaling law, with accuracy monotonically increasing as the number of examples within the context increases. Furthermore, it maintains general chat functionality, knowledge, and inference capabilities.
