KLEAR Textbook 교과서 Activity Tool 제작 프로젝트 - 프롬프트 프로젝트

더 프롬프트 컴퍼니는 2024년 1월부터 한국어 교육을 위한 AI Activity 툴과 챗봇을 제작하고 있습니다.

이 프로젝트는 미국 UH Press 출판사, University of Hawaii at Manoa, EALL의 한국어과가 함께하고 있습니다. 2024년 봄학기에는 하와이 대학교의 초급 한국어를 수강하는 학생을 대상으로 툴과 챗봇의 파일럿 테스트를 마쳤는데요, 1차 프로젝트에서는 AI 툴과 챗봇 48개를 제작했습니다.
프롬프트만으로 LLM의 한계를 보완하고 학생의 수준에 맞춘 의미있는 학습 도구를 제공할 수 있다는 성과가 있었습니다.

이 글에서는 1차 프로젝트에 대한 소개, 프로젝트 배경, 프로젝트 진행 방법, 설문 조사를 담은 연구 내용을 담았습니다. 2024년 미국 인디애나 주 Indiana University 에서 열린 제29회 AATK 학회에 채택되었습니다.

Abstract

Enhancing Korean Language Learning through AI and Chatbots: An In-Depth Study of Efficacy, User Experiences, and Challenges

The incorporation of Artificial Intelligence (AI) has become widespread in the field of language learning and teaching over the past decade. One form of AI, the chatbot, has gained popularity in language education for its ability to facilitate student learning in various aspects. This study focuses on evaluating the impact of these technologies on the Korean language learning experience, particularly at the beginner level. Its aim is to explore the effectiveness of generative AI in facilitating language acquisition, with a special emphasis on learners' engagement and perceptions.
Empirical studies by Zhang and Aslan (2021), Schmidt and Strasser (2022), Jeon (2021), and Aihua (2021), assert the potential benefits of AI chatbots in language learning. Language learning applications like Duolingo and Babbel utilize AI to provide personalized feedback, serving as valuable adjunct tools in the language learning experience. However, applying generative AIs to language learning presents unique challenges due to their peculiar characteristics. Therefore, this study developed AI tools and chatbots utilizing the advanced capabilities of ChatGPT-3.5 and ChatGPT-4 models, tailoring them specifically for Korean language learning contexts.

The current study involves the development of 48 unique AI tools and chatbots using advanced prompt engineering techniques. Designed to reinforce key language elements such as vocabulary, sentence structure, and verb conjugations, these chatbots were programmed to simulate real-world conversations, mirroring the scenarios presented in each textbook lesson. This functionality provides real-time feedback and correction on grammatical errors and vocabulary misuse. The primary aim is to create an immersive learning environment where learners can apply and test their language skills in practical, conversational contexts.

Using the pilot test of these chatbots, a detailed user survey was conducted with 50 learners at a university in the US who had prior experience in learning Korean. This survey sought to understand various aspects of their learning process, including their firsthand experiences after using the specifically chosen AI tools and chatbots for this survey.

According to the survey results, 70% of participants affirmed the utility of AI supplementary tools in their language progress. The conjugation tool, in particular, received high praise for its effectiveness in practicing grammar and vocabulary in a conversational setting. Notably, more than half of the participants acknowledged that AI tools positively influenced their understanding of Korean, especially in recognizing the subtle differences between Korean and English and appreciating the intricate details of language learning. However, the study also highlighted significant limitations and challenges. A notable issue was the occurrence of AI-generated inaccuracies or 'hallucinations', observed in about 20% of the cases. Participants reported instances where the AI provided incorrect sentence structures or tense forms, underlining its imperfection as a learning aid. The AI tools also struggled with nuances, specific contexts, and appropriate situational responses.

The findings suggest that while AI can be a powerful aid in language learning, educators and developers must address its current limitations and ensure learners are well-informed about these technologies.

Results and Insights:

1.

Positive Feedback:

70% of participants affirmed the utility of AI supplementary tools in their language learning progress. The conjugation tool was particularly praised for its effectiveness in practicing grammar and vocabulary in conversational settings.

2.

Improved Understanding:

More than half of the participants acknowledged that AI tools positively influenced their understanding of Korean. They noted an enhanced ability to recognize subtle differences between Korean and English and appreciated the intricate details of language learning facilitated by these tools.

3.

Challenges and Limitations:

Despite the positive feedback, significant limitations were also highlighted. Approximately 20% of the cases experienced AI-generated inaccuracies or 'hallucinations', where the AI provided incorrect sentence structures or tense forms. Additionally, the AI tools struggled with nuances, specific contexts, and appropriate situational responses.

The findings from this project suggest that while AI-powered tools can significantly aid language learning, it is crucial to address their current limitations. Educators and developers must ensure learners are aware of these limitations and provide guidance on how to use the AI tools effectively.

Methodology and Approach for creating AI-powered Activity Tools

Development Process:

1.

Selection of LLM Models:
The study utilized advanced capabilities of ChatGPT-3.5 and ChatGPT-4 models, chosen for their superior natural language processing capabilities.

2.

Prompt Engineering:
A total of 48 unique AI tools and chatbots were developed using advanced prompt engineering techniques. This process involved designing specific prompts that could elicit desired responses from the AI, ensuring that the tools could simulate real-world conversations and provide meaningful feedback to learners.

3.

Alignment with KLEAR Textbooks:

4.

This alignment ensured that the chatbots reinforced key language elements presented in the textbooks, such as vocabulary, sentence structure, and verb conjugations.

5.

Stimulation of Real-World Conversations:
The chatbots were programmed to mirror the scenarios presented in each textbook lesson, creating an immersive learning environment. They provided real-time feedback and correction on grammatical errors and vocabulary misuse, allowing learners to apply and test their language skills in practical contexts.

Challenging Points for Prompt Engineering

1.

단어 생성 컨트롤 Beginner-level words and sentences need to be generated, but LLMs cannot adjust the level. They often produce sentences that are too difficult for beginner Korean learners to understand.

2.

문장 레벨 컨트롤 Based on the textbook's dialogue scenarios, similar problems need to be generated for students, but control over this is lacking.

3.

일관성 유지와 맥락 유지 It was challenging to maintain consistency with the curriculum, and generate content that is appropriately tailored to the learners' proficiency levels.

Prompt Engineering 을 통한 해결방법

1️⃣ Few-shot prompt enginering

By using few-shot prompt engineering, I can explicitly instruct the LLM to generate content at a beginner level. For instance, you can include specific instructions in the prompt, such as "Create simple sentences with basic vocabulary suitable for beginner Korean learners." Additionally, you can provide examples of the type of sentences you want to ensure the model understands the desired complexity level. In the following example, I can restrict the set of words the LLM needs to generate.

Prompt Example. Lesson 2. 어때요?

“””You are an expert Korean language teacher. Your task is to generate extremely simple questions based on the {{형용사}} inputted by the user. 
Limit your response to less than 6 words. Respond in Korean. Include the English translation at the end of your response.

(Example) 
U: 맛있다.
A: 학교 식당 커피가 어때요? 맛있어요.
How is the school coffee? Delicious.
###
U: 많다.
A: 한국어 수업 숙제가 어때요? 재미있어요.
How is your Korean homework? Fun.
###
U: 좋다.
A: 요즘 기분이 어때요? 좋아요.
How are you feeling? Good.
###

{형용사}로 된 부분에는 Word Bank를 만들어 단어를 주어진 리스트 안에서만 생성하도록 했습니다. 레벨 밖 단어를 생성하지 못하도록 한 것인데, 제법 잘 컨트롤 합니다.

2️⃣ Chain-of-thought Prompting Technique

I found a specific problem that LLMs often produced sentences that were too difficult for beginner learners of Korean. I solved this issue with chain-of-thought prompt technique. Using this technique, you guide the model step-by-step to ensure it generates simple sentences suitable for beginners. By explicitly instructing the model to think through each step, the complexity of the output can be controlled. In the following example, you can check that tasks were breaking down into manageable steps and guiding the model through the process. In the output, it aligned with textbook content and scenarios.

Prompt Example. Lesson 2. Describing People” Practice Bot

"""
You are an expert Korean language teacher.
#STEP ONE:
After the user opens the conversation with "네!" or "Let's go!" or "I'm ready," your first task is to generate a table of information in Korean about an imaginary student. Your STEP ONE response should be in Korean. Do not include translations. Each field must not exceed 15 characters. Do not include any explanations. Do not respond in full sentences. Create a list of random characteristics that includes the following information:

**—
Name:
School Year:
Nationality:
Major:
Relationship (With user):
Hometown:
—**

##STEP TWO:

When the user responds with their paragraph, your task is to point out any grammar mistakes and offer a corrected version. Your grammar corrections should be in English, but keep the rest of the interaction in Korean. After you provide corrections, start the cycle over by offering a new unique set of characteristics. Do not repeat any of the information you generated previously. Your STEP TWO response should be in English for the corrections and Korean for the new table. Repeat this cycle until the user closes the conversation.

Do not use only Korean names, use other countries' names as well.

###### The following are examples of correct interactions. In these examples, U is the user and A is you:

**U: "네! Let's go!"**

**A:**
Name: 나타샤
School Year: 3학년
Nationality: 이탈리아 사람
Major: 경영학
Relationship (With user): 친구
Hometown: 로마

### STEP THREE: 
Now, please turn this table into a paragraph in Korean, and I'll check your grammar.**

**U: "제 친구의 이름은 나타샤예요. 나타샤는 3학년이에요. 나타샤은 이탈리아 사람잉에요. 나타샤의 전공은 경영학이에요. 나타샤는 로마에서 왔어요."**

- *A: "That looks great! Just a few notes:

—Grammar Mistakes:

1. 나타샤은 → 나타샤는
2. 사람잉에요 → 사람이에요

—Corrected Paragraph:
제 친구의 이름은 나타샤예요. 나타샤는 3학년이에요. 나타샤는 이탈리아 사람이에요. 나타샤의 전공은 경영학이에요. 나타샤는 로마에서 왔어요.

—New Set of Characteristics:
Name: 루카
School Year: 2학년
Nationality: 일본 사람
Major: 의학
Relationship (With user): 동료
Hometown: 도쿄"

"""

프롬프트 엔지니어링 전략

1.

Structure Prompts

LLM 이 이해하기 쉬운 형태로 프롬프팅을 구조화한 것이 포인트 입니다.

Heading 과 Bullet Points(해시태그 사용), Numbered lists 를 사용하여 문장의 요소를 구분 했습니다. AI가 이해하기 쉬운 형태로 task 를 전달하여, 목적에 맞는 결과를 얻을 수 있습니다.

1.

Iterative Refinement

LLM이 해야 할 task 단계를 세 개로 나누어, 반복적으로 처리하게 합니다. 이렇게 반복하게 하면, AI 답변에서 실수를 줄이고 잘못된 결과 출력을 막을 수 있습니다. 무엇보다 목표에 딱 맞는 결과를 얻을 수 있습니다.

1.

Contextual Relevance and Diversity

LLM의 목표를 정확하게 한정하여 문맥을 제어할 수 있습니다. Korean grammar를 가르치는 것을 목표로 하고, 프롬프트의 목적을 사용자의 문법 오류를 수정하고 설명하도록 했습니다. 또, 템플릿을 사용하여 같은 학습 레벨을 학습 할 수 있도록 했습니다.