Self-Consistency on Prompt
Self-consistency is an innovative technique that enhances the reasoning abilities of AI language models. It improves upon the traditional Chain of Thought (CoT) method by generating multiple reasoning paths and selecting the most consistent answer among them. This approach is effective for tackling complex arithmetic or commonsense logic problems.
This technique was proposed in the 2023 study <Self-Consistency Improves Chain of Thought Reasoning in Language Models> by Xuezhi Wang.
SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS.pdf823.42KB
How self-consistency operates
•
Generating multiple reasoning paths: By using several short CoT prompts, multiple reasoning paths are created.
•
Selecting the most consistent answer: Identify and choose the answer that is most consistent among the generated paths.
•
Aggregating results: Combine the various reasoning paths to arrive at a final answer.
How this differs from conventional CoT
The conventional CoT approach prompts the model to break down its reasoning into clear steps, rather than simply providing the answer. In contrast, self-consistency builds on this CoT process by generating multiple reasoning paths and selecting the most consistent outcome among them.
Practical usage
Let’s suppose we have another math problem. The problem is as follows:
"Younghee stopped twice while riding her bike for 60 km. She made the first stop after 20 km, and the second stop 15 km before finishing the trip. How far did Younghee travel between the first and second stops?"
Typically, this is what appears when you enter it into the prompt.
Younghee traveled a total distance of 60 km. After the first stop, she covered 60 - 20 = 40 km. So, the answer is 40 km.
That was incorrect again. On the surface, it may seem like a simple subtraction, but it turns out my LLM buddy didn’t find it easy. Let’s try solving it using the self-consistency model.
Inference Path 1: "Younghee traveled a total distance of 60 km. Her first stop was after 20 km, and the second was 15 km before the trip ended. Therefore, she traveled 60 - 20 - 15 = 25 km between her first and second stops. The answer is 25 km."
Inference Path 2: "On a 60 km trip, Younghee’s first stop was at 20 km, and the second at 60 - 15 = 45 km. The distance between the first and second stops is 45 - 20 = 25 km. So, the answer is 25 km."
Both of these reasoning paths arrive at the correct answer. This is precisely how the self-consistency technique works. For an even simpler example:
Q: Younghee took her first stop 20 km into her 60 km bike ride. How far did she travel after her first stop?
A: Younghee stopped after 20 km. So, the distance remaining after her first stop is 60 - 20 = 40 km.
Q: Younghee’s second stop was made 15 km before finishing her trip. How far into the trip was her second stop?
A: The second stop was made 15 km before the trip ended. So, the distance from the start to her second stop is 60 - 15 = 45 km.
Q: How far did she travel between the first and second stops?
A: After the first stop, 40 km remained, and the second stop was at the 45 km mark. Therefore, the distance between the two stops is 45 - 20 = 25 km.
Do you get the difference?
"Younghee stopped twice while biking 60 km. The first stop was after 20 km, and the second was 15 km before the trip ended. What is the distance Younghee traveled between the first and second stops?"
The problem is solved by splitting this sentence and responding to each part separately. This way, even problems that GPT-3, LLaMA2, Claude and others couldn't solve before can now be tackled. And if you use this together with the upcoming 'thought tree' method, you can partially overcome the gap in performance caused by the size of CoT models.
ⓒ 2023. Haebom, all rights reserved.
You may use this for commercial purposes, provided you credit the source and have the copyright holder's permission.
