This paper focuses on crowdsourcing techniques used to build large-scale refined datasets, which is one of the main driving forces for the development of artificial intelligence (AI) and machine learning (ML). Labels generated through crowdsourcing may contain noise for various reasons, which adversely affects the learning performance. In this paper, we introduce the latest research trends in learning from noisy crowdsourced labels. We review major crowdsourcing models and methodological treatments, from classical statistical models to recent deep learning-based approaches, and especially emphasize the connection with signal processing (SP) theory (such as the identifiability of tensor and non-negative matrix factorizations), suggesting new solutions to long-standing challenges in crowdsourcing. We also cover new topics that are important for the development of next-generation AI/ML systems, such as crowdsourcing with reinforcement learning and human feedback (RLHF), and direct preference optimization (DPO). In particular, we cover techniques that are important for fine-tuning large-scale language models (LLMs).
