Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models

Created by
  • Haebom

Author

Martina Miliani, Serena Auriemma, Alessandro Bondielli, Emmanuele Chersoni, Lucia Passaro, Irene Sucameli, Alessandro Lenci

Outline

In this paper, we introduce ExpliCa, a novel dataset for assessing explicit causal reasoning. ExpliCa integrates causal and temporal relations that are presented in various linguistic orders and explicitly expressed as linguistic connectives. It includes crowdsourced human acceptability ratings and is evaluated against seven commercial and open-source LLMs using prompting and perplexity-based metrics. Our results show that even the top-ranked models do not reach 0.80 accuracy, suggesting that models tend to confuse temporal and causal relations, and that performance is strongly affected by the linguistic order of events. We also find that perplexity-based scores and prompting performance are affected differently by model size.

Takeaways, Limitations

•
Takeaways: Introducing a new dataset, ExpliCa, for assessing explicit causal inference ability, enabling in-depth analysis of the causal inference ability of LLMs. It clearly presents the limitations of causal inference ability of LLMs, and reveals key issues such as confusion between temporal and causal relationships, and the influence of linguistic order. It provides new insights into the correlation between model size and performance.
•
Limitations: Further review of the size and diversity of the ExpliCa dataset is needed. Discussion of the prompting method and perplexity metric used in the evaluation is needed. Further research is needed on more diverse types of causal inference problems and complex sentence structures.
👍
Made with Slashpage