Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

The StudyChat Dataset: Student Dialogues With ChatGPT in an Artificial Intelligence Course

Created by
  • Haebom

Author

Hunter McNichols, Fareya Ikram, Andrew Lan

Outline

This paper introduces StudyChat, a real-world college student dataset collected through interactions with a tutoring chatbot based on a large-scale language model (LLM). We collected 16,851 interactions with an LLM-based web application while students were completing programming assignments in a college-level AI course and annotated them using a conversational act labeling technique. Data analysis revealed that students who used the LLM for conceptual understanding and coding support performed better on assignments and exams, while those who used it for report writing or to avoid assignment learning objectives performed worse on exams. StudyChat can serve as a shared resource for further research on the role of LLM in education.

Takeaways, Limitations

Takeaways:
Providing a real-world dataset (StudyChat) for analyzing LLM-based learning tool usage patterns.
Presentation of correlation analysis results between LLM utilization methods and academic achievement (concept understanding and coding support vs. report writing and assignment avoidance).
Contribute to developing strategies for effectively utilizing LLM in education.
Laying the foundation for further research on the use of LLMs in education.
Limitations:
The StudyChat dataset was collected from students in an AI course at a specific university, so there are limitations to its generalizability.
The analysis results show a correlation, but do not clearly establish causality.
It is possible that the diversity and complexity of how LLMs are utilized may not be fully captured.
The web application replicates the functionality of ChatGPT and is not ChatGPT itself.
👍