Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Evaluating User Experience in Conversational Recommender Systems: A Systematic Review Across Classical and LLM-Powered Approaches

Created by
  • Haebom

Author

Raj Mahmud, Yufeng Wu, Abdullah Bin Sawad, Shlomo Berkovsky, Mukesh Prasad, A. Baki Kocaballi

Outline

This paper systematically reviews 23 empirical studies published from 2017 to 2025 according to the PRISMA guidelines to analyze the limitations of existing research on the user experience (UX) evaluation of conversational recommender systems (CRSs) and suggest future research directions. Specifically, we point out the paucity of research on the UX evaluation of adaptive CRSs and large-scale language models (LLMs). We analyze UX concept definitions, measurement methods, domains, adaptability, and the influence of LLMs. We uncover shortcomings such as the dominance of follow-up research, the rare evaluation of turn-based emotional UX components, and the rare connection between adaptive behaviors and UX outcomes. We also highlight the epistemological opacity and verbosity of LLM-based CRSs. We propose a structured UX metric synthesis for the development of more transparent, engaging, and user-centered CRS evaluation practices, a comparative analysis of adaptive and non-adaptive systems, and a future-oriented agenda for UX evaluation that considers LLMs.

Takeaways, Limitations

Takeaways:
We provide a systematic analysis of the user experience (UX) evaluation of conversational recommender systems (CRSs).
Clarify Limitations on UX evaluation of adaptive CRS and LLM-based CRS.
We present structured UX metrics and a forward-looking agenda for developing more transparent and user-centric CRS evaluation practices.
Highlights the specific UX issues of LLM-based CRS (epistemological opacity, lengthiness).
Limitations:
Points out the limitations of UX evaluation methods that rely on follow-up research.
There is a lack of evaluation of turn-by-turn emotional UX components.
There is insufficient analysis of the link between adaptive behavior and UX outcomes.
Lack of consideration of epistemological opacity and lengthiness in UX evaluations of LLM-based CRSs.
👍