개발 일지
Langchain으로 RAG 구현하기
Y
yeji Kim
Fine-tuning - PEFT, QLoRa
•
더 발전된 모델인 Fusion-in-Decoder(FiD)나 Atlas를 고려하기
•
키워드 검색과 벡터 검색을 함께 사용하기
•
Vector DB 대신 Knowledge Graph를 사용하기
Python langchain
chunk_size
chunk_overlap
chunk_overlap
splitter = CharacterTextSplitter.from_tiktoken_encoder(
separator="\n",
chunk_size=500,
chunk_overlap=50
)
separator="\n",
chunk_size=500,
chunk_overlap=50
)
docs = data_loader.load_and_split(text_splitter=splitter)
embeddings = OpenAIEmbeddings()
cached_embeddings = CacheBackedEmbeddings.from_bytes_store(embeddings, cache_dir)
embeddings = OpenAIEmbeddings()
cached_embeddings = CacheBackedEmbeddings.from_bytes_store(embeddings, cache_dir)
과정
1.
Raw data → connecting
2.
Connecting → embedding
3.
Embedding
4.
Vector DB
5.
Retrieval
6.
Subscribe to '아무튼-작업일지'
Subscribe to my site to be the first to receive notifications and emails about the latest updates, including new posts.
Join Slashpage and subscribe to '아무튼-작업일지'!
Join Slashpage and subscribe to '아무튼-작업일지'!
