The era of small and strong LLM will come someday?!
Haebom
Research in language models shows that small models can play an important role, not just large ones. Researchers at Microsoft trained small language models based on children's stories and found that these small models could quickly learn grammar and coherence.
On the one hand, some argue that MS's model has no real utility, but rather has a high evaluation score due to biased learning on a specific dataset. Even leaving aside moral and ethical issues, a small but high-performance model is actually everyone's dream. This is especially true in places where infrastructure and financial constraints are limited.
There are many attempts to quickly serve embedding models on foundation models of 13B to 60B or more by placing them on top of foundation models of 100B or more, but they all ultimately expect to obtain great utility at low cost. This is almost the only method that players in this market can choose, excluding Google, OpenAI, and Meta, which have already secured infrastructure, data sets, and funding.
Small language models can be trained on small data sets rather than large ones.
Training smaller models uses fewer resources and is therefore accessible to more researchers.
Subscribe to 'haebom'
📚 Welcome to Haebom's archives. --- I post articles related to IT 💻, economy 💰, and humanities 🎭. If you are curious about my thoughts, perspectives or interests, please subscribe. Would you like to be notified when new articles are posted? 🔔 Yes, that means subscribe. haebom@kakao.com
