This paper focuses on the inference-intensive regression (RiR) problem. RiR is the process of inferring nuanced numerical features from text. Unlike standard language regression tasks like sentiment analysis or similarity measurement, RiR often appears in ad hoc problems like rubric-based scoring or domain-specific retrieval. Its unique characteristic is that it requires in-depth text analysis in situations where limited task-specific training data and computational resources are available. We define three practical problems as RiR tasks, establish a baseline, and test the hypothesis that fine-tuning a fixed large language model (LLM) prompt and a Transformer encoder would be challenging in RiR. To address this, we propose MENTAT, a lightweight method that combines batch-aware prompt optimization and neural network ensemble learning. We demonstrate that MENTAT achieves up to 65% performance improvement over baseline models. However, we conclude that significant room for improvement remains in the field of RiR.
