Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Sign Spotting Disambiguation using Large Language Models

Created by
  • Haebom

Author

JianHe Low, Ozge Mercanoglu Sincan, Richard Bowden

Outline

This paper presents a novel framework for sign spotting, which identifies and localizes individual sign signs in continuous sign language videos to address the data shortage problem in sign language translation. To address the lexical flexibility and ambiguity issues of existing sign spotting methods, we propose a non-training approach that integrates a large-scale language model (LLM). We extract spatiotemporal and hand features and match them against a large sign language dictionary using dynamic temporal warping and cosine similarity. We then leverage the LLM to perform context-sensitive lexical disambiguation using beam search. Experimental results on synthetic and real-world sign language datasets demonstrate improved accuracy and sentence fluency compared to existing methods.

Takeaways, Limitations

Takeaways:
We demonstrate that LLM can be used to improve the accuracy and sentence fluency of sign language sign detection.
Increase vocabulary flexibility and reduce the need for model retraining through a learning-free framework.
Contribute to solving the problem of data shortage by effectively utilizing a large-scale sign language dictionary.
Leveraging LLM's context-aware capabilities to mitigate ambiguity in sign language sign discovery.
Limitations:
The performance of the proposed method may depend on the performance of LLM.
The quality of a large sign language dictionary can have a significant impact on the results.
Further validation of generalization performance in real-world complex sign language video environments is needed.
Reliance on datasets restricted to specific languages or sign language styles.
👍