Overcoming the data shortage problem through synthetic data generation
Synthetic data generation is a technology where artificial intelligence creates virtual data on its own when real data is insufficient. Based on real existing data, AI can 'synthesize' new data, which can be used to learn even more.
For example, when creating a chatbot that needs to understand legal documents from country A, if there aren't enough actual legal documents from that country, you can generate new legal documents derived from the legal documents of other countries using synthetic data generation. These documents may not exist in reality, but they can be used for training the chatbot to better understand legal documents.
Synthetic data like this helps save time and cost when training AI models and makes it possible to prepare for a variety of situations. It also provides the flexibility to adjust the data to specific domains or languages.
RAG (Retrieval-Augmented Generation) is a model that searches for information and generates answers based on what it finds. Generating synthetic data for RAG can be used to build a database that RAG can retrieve from. Using synthetic data, the RAG model learns how to generate answers to a wide range of questions and provide more accurate and useful information in response to real users' queries.
However, ensuring the quality of synthetic data is an important challenge. If the data does not accurately reflect real situations, the model may end up learning incorrect information. Therefore, it's essential to guarantee the diversity and quality of the data when generating synthetic datasets, and to regularly evaluate them to make sure the model can give answers that are suitable in real world scenarios. During this process, it’s important to check whether the data covers various scenarios relevant to actual work, and to continue improving based on the model’s performance.
Any real-world use cases?
•
Study by Dai et al. (2022): This research suggests a method that achieves near state-of-the-art performance using only 8 manually labeled examples and a large pool of unlabeled data (like all parsed legal documents).
•
Use of synthetic data in machine learning models: Machine learning models trained with synthetic data can sometimes outperform models trained with real data in certain situations. This can help researchers identify cases where using synthetic data for training is preferable, and it can sidestep issues like bias, privacy, security, and copyright concerns that affect real datasets.
•
Development of synthetic data by MOSTLY AI: MOSTLY AI is at the forefront of generating synthetic data for AI model development and software testing. This shows rapid progress in the AI and synthetic data field. (Link) Of course, this is just a claim—I’m not sure it actually works. It does seem like promotional material, but that's what they're saying.
•
In certain industries, such as finance or healthcare, there can be legal or ethical barriers to obtaining real data. As a result, there is a strong demand in these fields to generate the needed training data based on data that has already received consent.
In conclusion, synthetic data generation is extremely helpful for quickly developing and testing AI models when data is scarce, and it is especially crucial for models like RAG to generate better answers based on retrieved information. Typical examples include tools like GPTs, Bing, Google's Bard, or Notion Q&A. These systems generate improved answers based on documents or files uploaded or written by the user.
ⓒ 2023. Haebom, all rights reserved.
Commercial use is allowed with the copyright holder's permission and proper source citation.
