Post-training of large-scale language models (LLMs) is essential for aligning pre-trained language models (PLMs) with human preferences and downstream tasks. While PLMs typically exhibit well-calibrated confidence, post-trained language models (PoLMs) often exhibit overconfidence, assigning high confidence to both correct and incorrect responses, which can compromise their reliability in critical applications. A major obstacle to calibrating PoLMs is the lack of labeled data for individual downstream tasks. To address this, this paper proposes Disagreement-Aware Confidence Alignment (DACA), a novel unsupervised learning method that optimizes parameters (e.g., temperature $\tau$) in post-trained confidence alignment. This method is motivated by the underconfidence problem caused by prediction discrepancies between the PLM and PoLM due to temperature adjustment. Theoretically, the PLM's confidence underestimates the PoLM's prediction accuracy for discordant examples, leading to larger $\tau$ and underconfidence predictions. DACA mitigates this by selectively using only consensus examples for calibration, effectively isolating the impact of inconsistencies. In this way, DACA prevents excessive $\tau$ in temperature regulation caused by inconsistent examples, thereby improving calibration performance. Extensive experiments have demonstrated that DACA improves the average ECE of open-source and API-based LLMs (e.g., GPT-4o) by up to 15.08% on common benchmarks.
