[공지사항]을 빙자한 안부와 근황 
Show more

Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

HATS: Hindi Analogy Test Set for Evaluating Reasoning in Large Language Models

Created by
  • Haebom

Author

Ashray Gupta, Rohan Joseph, Sunny Rai

Outline

In this paper, we present a novel dataset, Hindi Analogy Test Set (HATS), for assessing analogy performance in Hindi. HATS consists of 405 multiple-choice questions taken from Indian government exams and is used to assess the analogy performance of various language models. In this paper, we evaluate state-of-the-art multilingual LLMs using various prompting strategies and a grounded Chain of Thought approach based on cognitive theory and suggest a method to improve model performance on Hindi analogy tasks. Experimental results show that model performance is best when English prompts are used, regardless of prompting strategy. This study addresses the critical lack of resources for assessing LLM reasoning performance in Hindi.

Takeaways, Limitations

•
Takeaways:
◦
We provide a new dataset HATS for evaluating LLM reasoning ability in Hindi language.
◦
We demonstrate that the grounded Chain of Thought approach contributes to improving model performance on the Hindi inference problem.
◦
It contributes to assessing the analogical ability of multilingual LLMs and analyzing their ability to generalize across languages.
◦
We examine the effectiveness of English prompts and suggest directions for future research.
•
Limitations:
◦
Currently, HATS is limited to Indian Government Examination questions and may not fully reflect the diverse types of analogies in the Hindi language.
◦
The type and size of LLM used for evaluation may be limited.
◦
The finding that English prompts are most effective suggests the need to consider cross-linguistic bias.
◦
Further research is needed to explore the generalizability of the grounded Chain of Thought approach.
👍
Made with Slashpage