Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Comparing Human and Language Models Sentence Processing Difficulties on Complex Structures

Created by
  • Haebom

Author

Samuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant

Outline

This paper systematically compares large-scale language models (LLMs) capable of human conversation to determine whether they experience similar comprehension difficulties to humans. Sentence comprehension data from humans and five state-of-the-art LLMs on seven challenging linguistic structures were collected and analyzed in a unified experimental environment. In particular, LLMs struggle with the sentence "Garden Path," and model performance correlates with human performance in proportion to the number of parameters. Furthermore, performance differences between complex and basic structures are similar for both humans and LLMs, with convergence and divergence observed depending on the model's strength.

Takeaways, Limitations

LLM provides new insights into sentence comprehension skills, particularly the ability to process complex language structures.
Identifying LLM vulnerabilities for specific structures, such as the 'garden pass' sentence.
Analysis of performance changes and human-like similarity according to model size
Models that are too weak or too strong can reduce the performance gap with humans.
The limited range of language structures used in the experiment
Limitations in the generalizability of performance results for specific LLM models.
Lack of performance verification in real-world use environments for LLM
👍