English
Share
Sign In
Active Prompts: Boost Performance with Rich Examples
Active-Prompt has several important differences from traditional CoT methods. While traditional CoT methods rely on fixed, human-annotated examples, Active-Prompt dynamically applies specific example prompts to different tasks. Active-Prompt follows these steps:
1.
Uncertainty Estimation: Using a large-scale language model (LLM), questions for a specific task are asked many times to generate multiple possible answers, and uncertainty metrics are used to calculate the uncertainty of each question. Uncertainty metrics used include the number of inconsistent answers, entropy, and variance.
2.
Selection: Select the most uncertain questions for annotation based on their computed uncertainty.
3.
Annotation: Human annotators provide reasoning and answers to selected questions, creating new examples.
4.
Inference: Perform inference for each question using the newly annotated examples and select the most consistent answer.
Active Prompting with Chain-of-Thought for Large Language Models.pdf937KB
This process is in contrast to existing CoT methods that rely on a fixed set of examples and human annotations, where Active-Prompt dynamically selects and annotates the most informative questions by measuring uncertainty to maximize the performance of the LLM. This is particularly suitable for complex inference tasks, and has been shown to improve the adaptability and accuracy of the model across a wide range of tasks.
"영희가 장거리 하이킹을 위해 백팩을 싸고 있는데, 그녀는 무엇을 가져가야 할까요?"The language model generates multiple inference paths and answers to this question. For example, it can generate a list of items needed for a hike, or recommend equipment needed for a specific situation. Among the multiple answers generated, the one with high uncertainty is selected and refined by a human annotating and refining it along with the inference process.
"철수가 10층짜리 호텔에서 방을 예약했습니다. 각 층에는 동일한 방이 10개씩 있는데, 모든 방을 사용할 수 있을까요?"The language model generates multiple answers to this question, and selects the least certain answer. For example, the model might answer, “The last floor is unavailable, so there are 90 rooms available.”
The core of the Active-Prompt method is to find the most uncertain answer among the many answers generated by the language model and reinforce it with human annotations to obtain more accurate and reliable results .
ⓒ 2023. Haebom, all rights reserved.
It may be used for commercial purposes with permission from the copyright holder, provided the source is cited.
