This paper presents a model to improve the efficiency of decision-making systems that combine artificial intelligence (AI) and humans, particularly social media content moderation systems. Existing AI-human pipelines rely on simple threshold-based heuristics that fail to account for AI risk estimation uncertainty, temporal variability in content inflow, human review capacity, and selective sampling. In this paper, we propose a model in which AI observes contextual information to make classification and review decisions and schedules review tasks while accounting for delays in the human review system. During the human review process, AI errors are corrected and new data is acquired, with the goal of minimizing the cost of misclassified tasks. We present a suboptimal learning algorithm that carefully balances the classification loss of selectively sampled datasets, the inherent loss of unreviewed tasks, and the delay loss due to congestion in the human review system. Numerical experiments using an online content dataset demonstrate that our model significantly reduces misclassifications compared to existing methods. These results represent the first demonstration of online learning in a contextual queuing system.
