This paper addresses the vulnerabilities of using large-scale language models (LLMs) for data fitting and prediction generation. While LLMs demonstrate competitive predictive performance across a variety of tasks, we find that they are vulnerable to task-irrelevant changes in data representation (e.g., variable renaming). This phenomenon occurs in both in-context learning and supervised fine-tuning, as well as in both close-weight and open-weight LLMs. Analysis of the attention mechanism in open-weight LLMs reveals that they over-focus on tokens in specific positions. Even state-of-the-art models like TabPFN, specifically trained for data fitting, are not immune to these vulnerabilities. Therefore, current LLMs lack even a basic level of robustness to be used as a principled data fitting tool.
