This paper points out that although inference language models that enable multi-level inference via reinforcement learning have achieved state-of-the-art performance on several benchmarks, they are still vulnerable to hallucination phenomena, which confidently present incorrect answers. Therefore, it is important to quantify the confidence of the model in order to safely deploy the inference model in real-world applications. To this end, the paper explores uncertainty quantification of inference models and aims to answer three questions: whether the inference model is calibrated, how deeper inference affects model calibration, and whether models can improve calibration by explicitly inferring their inference process. Using introspective uncertainty quantification (UQ), we evaluate state-of-the-art inference models on various benchmarks. We find that inference models are generally overconfident, especially for incorrect answers, their self-expressed confidence estimates often exceed 85%, and that deeper inference exacerbates overconfidence, and that self-reflection can improve calibration (e.g., o3-Mini and DeepSeek R1), but not consistently across all models (e.g., Claude 3.7 Sonnet). Finally, we design essential UQ benchmarks and suggest important research directions to improve the calibration of inference models.
