As language models continue to develop, various experiments are being conducted. Among them, Retrieval Augmented Generation (RAG) is attracting attention as a technology that allows language models to understand and generate various contexts. However, RAG also has limitations. A new study has emerged to overcome these limitations, and it is the 'Self-Reflective Retrieval-Augmented Generation (SELF-RAG)'announced by the University of Washington and IBM research team.
What is Retrieval Augmented Generation (RAG)?
RAG stands for Retrieval Augmented Generation, a technology that allows language models to reference external databases or documents to generate more accurate and richer answers when generating answers to specific questions or requests.
Understanding with a restaurant example
For example, when a customer asks, “What is the most popular menu item at this restaurant?”, a typical language model can only answer based on previously learned data and cannot provide real-time information.
However, a language model using RAG can refer to a restaurant's reviews or menu data in real time and answer, "The most popular menu item at this restaurant right now is 'steak'."
Limitations of RAG
Indiscriminate information retrieval: Although RAG references external data, it lacks the ability to judge how accurate and reliable that information is.
Limited generalizability: RAG is unlikely to provide information optimized for a specific domain or situation. In other words, it may provide only too general information.
Understanding with a restaurant example
Indiscriminate information search: When RAG refers to restaurant reviews to recommend popular dishes, it does not know how recent and reliable the reviews are. For example, it may refer to reviews from several years ago to recommend old popular dishes.
Limited Universality: When a customer says, "I'm a vegetarian, can you recommend some dishes that I should try at this restaurant?", RAG may only recommend popular dishes that are generally popular and may not consider whether they are suitable for vegetarians.
To overcome these limitations, new techniques such as SELF-RAG are being studied, which help language models ‘self-reflect’ their own answers to generate more accurate and contextually relevant answers.
Features and operating principles of SELF-RAG
SELF-RAG (Self-Reflective Retrieval-Augmented Generation) is a technology developed to overcome the limitations of existing RAG. This model uses special tokens called "reflection tokens" to evaluate the quality of the text it generates. These tokens serve to evaluate how factual and relevant the generated text is, and what its overall quality is. In addition, this model provides sources for the generated answers to increase reliability.
What are Reflection Tokens?
"Reflection Tokens" are special tokens introduced in the SELF-RAG model. These tokens serve to automatically evaluate the quality of the text generated by the model. In other words, they evaluate how factual, relevant, and overall quality the generated text is. This allows the model to provide more accurate and reliable information.
Understanding with a restaurant example
Menu Recommendation: For example, let's say a customer asks, "What is the most popular menu item at this restaurant?" A typical RAG model will generate an answer by referencing various reviews or menu data. However, it is difficult to determine which information is more accurate or up-to-date.
Role of Reflection Tokens: The SELF-RAG model uses "Reflection Tokens" to evaluate the quality of the answers it generates. For example, it could generate an answer like "The most popular menu item at this restaurant is steak. (Source: Restaurant Review, October 2023)", where the "Source: Restaurant Review, October 2023" part is information added by the Reflection Token. Allergy Information: When a customer asks, “Does this menu contain nuts?”, the SELF-RAG model can find information about nut allergies and reply, “This menu contains nuts. If you have allergies, be careful. (Source: Restaurant Menu Information).” Again, “Source: Restaurant Menu Information” is generated by the Reflection Token.
Understanding with a restaurant example
Providing accurate information: When SELF-RAG recommends popular menu items, it uses “reflection tokens” to evaluate the reliability of the information by referring to various reviews and data. For example, it can ignore old or low-starred reviews and recommend menu items based on recent and high-starred reviews.
Contextual Response: When a customer tells SELF-RAG they are vegetarian, SELF-RAG can reflect this information by recommending only vegetarian options and even providing the source for that recommendation.
Performance metrics
SELF-RAG outperforms existing RAGs and other language models in several performance metrics, especially on the Open-domain QA dataset and the Long-form QA dataset (ASQA).
Subscribe to 'haebom'
📚 Welcome to Haebom's archives. --- I post articles related to IT 💻, economy 💰, and humanities 🎭. If you are curious about my thoughts, perspectives or interests, please subscribe. Would you like to be notified when new articles are posted? 🔔 Yes, that means subscribe. haebom@kakao.com
