English
Share
Sign In
Self-Consistency on Prompt
Self-consistency is a new technique that improves the reasoning ability of AI language models. It improves the existing Chain of Thought (CoT) method to generate multiple reasoning paths and select the most consistent answer among them. This method is effective for complex arithmetic reasoning or common sense logic problems.
This technique was proposed by Xuezhi Wang in his 2023 study, <Self-Consistency Improves Chain of Thought Reasoning in Language Models>.
SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS.pdf843KB
How self-consistency works
Generate multiple inference paths: Generate multiple inference paths using multiple short CoT prompts.
Select consistent answers: Identify and select the most consistent answer among the generated paths.
Aggregate results: Integrate multiple inference paths to arrive at a final answer
Differences from the existing CoT method
The CoT approach induces the model to explain the reasoning process step by step instead of giving a direct answer. On the other hand, self-consistency is extended by generating multiple reasoning paths based on this CoT approach and selecting the most consistent result among them.
How to use it in practice
Let's say we have another math problem. The problem is as follows:
"Younghee stopped twice during her 60km bike ride. The first stop was after 20km and the second stop was 15km before the end of the trip. How far did Younghee travel between the first and second stops?"
Normally, when you type into the prompt, it will look like this:
Younghee traveled a total of 60 km. After the first stop, she traveled 60 - 20 = 40 km. Therefore, the answer is 40 km.
Again, I got it wrong. It may seem like a simple subtraction problem, but it wasn't easy for my LLM friend. Let's try to solve this with a self-consistent model.
Inference Path 1: "Younghee traveled a total of 60 km. The first stop was 20 km later and the second stop was 15 km before the end of the trip. Therefore, she traveled 60 - 20 - 15 = 25 km between the first and second stops. The answer is 25 km."
Inference path 2: "During a 60 km trip, the first stop was made at 20 km and the second stop was made at 60 - 15 = 45 km. The distance between the first and second stops is 45 - 20 = 25 km. The answer is 25 km."
Both of these inferences are correct. This is the self-consistency technique used. Here is an easier example.
Q: Younghee made her first stop 20km into her 60km bike ride. How far did she travel after her first stop?
A: Younghee made her first stop 20km later. Therefore, the remaining distance after the first stop is 60 - 20 = 40km.
Q: Younghee's second stop was 15km before the end of the trip. How far is the second stop in the total trip distance?
A: The second stop was 15km before the end of the trip. Therefore, the distance from the total trip distance to the second stop is 60 - 15 = 45km.
Q: What is the distance between the first and second stops?
A: The distance remaining after the first stop is 40 km, and the distance to the second stop is 45 miles. Therefore, the distance between the two stops is 45 - 20 = 25 km.
Do you see the difference?
"Younghee stopped twice during her 60km bike ride. The first stop was after 20km and the second stop was 15km before the end of the trip. How far did Younghee travel between the first and second stops?"
This is a problem that is solved by segmenting the sentence and answering each answer separately. By doing this, you can see that problems that GPT-3, LLaMA2, Claude, etc. could not solve are solved, and if used together with the thought tree technique that will be released later, it will be possible to overcome the gap in model size, which was a shortcoming of CoT, to some extent.
ⓒ 2023. Haebom, all rights reserved.
It may be used for commercial purposes with permission from the copyright holder, provided the source is cited.