Will the age of compact, powerful LLMs come someday?!

Haebom

Oct 7, 20232y ago

Language model research shows that not only large models but also small models can play an important role. Microsoft researchers trained small language models based on children's stories and discovered that these small models can quickly acquire grammar and consistency.

Meanwhile, some argue that Microsoft's model simply scored highly by overfitting to a specific dataset and has no real utility. Putting aside moral and ethical issues, a small yet high-performing model has become something everyone dreams of. This is especially true for those with infrastructure or financial limitations.

Research is actively being conducted on models ranging from 13B to 60B parameters, and there are also various efforts to quickly deploy embedding models on top of foundation models with over 100B parameters. Ultimately, all these approaches aim to achieve maximum utility at minimal cost. Excluding Google, OpenAI, and Meta, who've already secured their infrastructure, datasets, and funding, this is virtually the only option left for other players in the market.

Effective Small Language Models: Microsoft’s 1.3 Billion Parameter phi-1.5 - KDnuggets

Learn about Microsoft’s 1.3 billion parameter model that has outperformed Llama 2’s 7-billion parameters model on several benchmarks.

kdnuggets.com

Key points

•

Small language models can be trained on small data sets instead of large ones

•

Training small models requires less resources, making them accessible to more researchers.

Subscribe to 'haebom'

📚 Welcome to Haebom's archives.
---
I post articles related to IT 💻, economy 💰, and humanities 🎭.
If you are curious about my thoughts, perspectives or interests, please subscribe.
haebom@kakao.com