Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages

Created by
  • Haebom

Author

David Demitri Africa, Suchir Salhan, Yuval Weiss, Paula Buttery, Richard Diehl Martinez

Outline

The authors study whether pretraining a small-scale decoder language model (LM) in an environment with limited memory and latency enables rapid adaptation to unseen languages and zero-shot transfer. Specifically, they use a method that replaces some of the model's autoregressive objectives with first-order Model-Agnostic Meta-Learning (MAML). Experiments are conducted on Tagalog and Cebuano, and they demonstrate that MAML improves zero-shot micro-F1 scores and reduces convergence times.

Takeaways, Limitations

Improving Zero-Shot NER Performance of Small-Scale Decoder LM Using MAML
Suggesting possibilities for solving the NER problem in low-resource languages such as Tagalog and Cebuano.
2-6% micro-F1 improvement when tuning only the head, 1-3% when tuning the entire head
Reduce convergence time by up to 8%
Emphasize the importance of surface anchors (single-token person entities appearing with the Tagalog case particles si/ni)
The number of languages tested is limited to two.
Experiments were conducted on various model sizes (11M - 570M)
👍