Category
- ✏️ Research
Date of issue
Jan 22, 2024
Generative AI Trends in 2023
: Small and efficient models, LoRA, MoE, and multimodal
Generative AI is undergoing an exciting transformation in 2023. While large-scale models dominated the field in the past, this year marks the emergence of small, efficient models like Low-Rank Adaptation (LoRA) and Mixture of Experts (MoE), as well as the expansion of the Large Language Model (LLM) domain.
1. The emergence of small and efficient models
1.1. Why do we need a small and efficient model?
In the past, large-scale models with hundreds of billions or trillions of parameters dominated the market. However, these models were prohibitively expensive to train and required significant memory and computing power, making them difficult to apply to real-world applications.
This has led to the emergence of compact and efficient models. These models have the advantage of being inexpensive to train and easy to run on a variety of devices.
1.2. Representative examples of small and efficient models
•
Ⓜ️Mistral: 7B model outperforms larger models in most benchmarks
•
🅿️Phi (Microsoft) series: With parameters of only 1.3B to 2.7B, it is much smaller but offers impressive performance.
•
🅾️Orca 2 (Microsoft): Extracts knowledge from larger models like GPT-4 into smaller models, targeting inference methods like stepwise analysis and recall, rather than simple imitation learning. Orca 2 outperforms similarly sized models, competing with models 5-10 times larger.
2. The emergence of LoRA (Low-Rank Adaptation), which helps with efficient fine tuning.
LoRA allows for parameter-efficient fine-tuning, significantly reducing computational load and storage requirements. This saves resources and enables user customization for a variety of applications without compromising the core functionality of the underlying model.
•
The Mixture of Experts Explained (MoE) approach also represents a significant leap forward in LLM architecture in 2023.
◦
MoE improves performance for complex tasks that may be difficult to handle with a single model.
◦
MoE simplifies complex problems by breaking them down into smaller, more manageable subproblems. Each subproblem is solved by a specialized submodel (expert). This improves the performance of the overall model.
3. Expansion of the LLM domain (multimodal)
LLM has evolved from a general foundational model to expand its capabilities beyond language processing. It can now understand and generate text, code, visual content, audio, and more.
•
Use Cases for Multimodal LLM
◦
Chatbots that utilize text and images together: Provide a more natural and richer user experience than text-based chatbots.
◦
Automatically generate video subtitles: Understand the video content and automatically generate subtitles.
◦
Voice Command Recognition and Response: Developing a More Accurate and Natural Voice Recognition and Response System
4. Conclusion
In 2023, generative AI will usher in a new trend with the emergence of small, efficient models. This trend is expected to further develop and be utilized in a variety of fields.
Reference link
Subscribe to 'The Invention Lab Blog'
If you subscribe to the site, you can be the first to receive the latest updates, including new posts, via notifications and email.
Join Slashpage and subscribe to 'The Invention Lab Blog'!
Join Slashpage and subscribe to 'The Invention Lab Blog'!
