This paper highlights the impact of AI system prediction accuracy and reliability of confidence estimates on decision quality in situations where AI inputs are used for human decision-making. We highlight the role of AI's metacognitive sensitivity—the ability to assign confidence scores that accurately distinguish between correct and incorrect predictions—and present a theoretical framework for evaluating the combined impact of AI's prediction accuracy and metacognitive sensitivity in hybrid decision-making environments. Our analysis identifies conditions under which AI with low prediction accuracy but high metacognitive sensitivity can improve the overall accuracy of human decision-making. Finally, behavioral experiments confirm that higher AI metacognitive sensitivity improves human decision-making performance. These results highlight the importance of evaluating AI support not only for accuracy but also for metacognitive sensitivity, and optimizing both to achieve superior decision-making outcomes.
