Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Discerning minds or generic tutors? Evaluating instructional guidance capabilities in Socratic LLMs

Created by
  • Haebom

Author

Ying Liu, Can Li, Ting Zhang, Mei Wang, Qiannan Zhu, Jian Li, Hua Huang

Outline

The conversational capabilities of large-scale language models (LLMs) offer significant opportunities for scalable and interactive tutoring. Existing research has focused on Socratic question generation, but has overlooked the crucial aspect of adaptive guidance based on the learner's cognitive state. This study shifts focus beyond question generation to instructional guidance. We question whether LLMs can mimic expert tutors, who dynamically adjust their strategies based on the learner's state. To this end, we propose GuideEval, a benchmark based on real-world educational conversations, and evaluate instructional guidance through a three-step behavioral framework: (1) recognition (inferring learner state), (2) accommodation (adapting teaching strategies), and (3) prompting (stimulating appropriate reflection).

Takeaways, Limitations

Takeaways:
We found that LLMs struggle to provide effective adaptive scaffolding based on learner status.
The guidance performance was significantly improved by introducing a behavioral guidance fine-tuning strategy.
The assessment paradigm shifted from content-centered assessment to learner-centered, state-aware interaction.
Limitations:
Traditional LLMs often fail to provide effective adaptive scaffolding when learners become confused or need to reorient themselves.
Analysis of failure cases provides an intuitive understanding of these shortcomings.
👍