전체 글 (198) 썸네일형 리스트형 (Docker) ollama + chroma로 RAG 구성 RAG 구성 with docker 환경 정의환경 변수>>> CHROMA_IP = '***'>>> CHROMA_PORT = 8000>>> OLLAMA_IP = '***'>>> OLLAMA_PORT = 11434>>> LLM_MODEL = 'llama3.1'ollama / chromadb 모두 docker로 구성됨 Vector store 구성 및 Document embeddingChromadb client 정의>>> import chromadb>>> clientChroma = chromadb.HttpClient(host=CHROMA_IP, port=CHROMA_PORT)>>> print(clientChroma.list_collections())chromadb 접속을 위한 client 정의list_colle.. LangChain - LLM Parameter from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate prompt = ChatPromptTemplate.from_messages([ ("system", "This system can response about IT system"), ("user", "{user_input}"), ]) params = { "temperature" : 0.3, "max_tokens" : 100, "frequency_penalty" : 0.5, "presence_penalty" : 0.5, "stop" : ["\n"] } model = ChatOpenAI(max_tokens=500) query = 'Pacemake.. LangChain - 2.6 Retrievers 참조 : https://python.langchain.com/docs/modules/data_connection/retrievers/ Retrievers | 🦜️🔗 Langchain A retriever is an interface that returns documents given an unstructured query. It is more general than a vector store. python.langchain.com Retrievers 구조화 되지 않은 쿼리가 제공되면 문서를 반환하는 Interface로 문서 저장 없이 반환하기만 함 Name Index Type Uses an LLM When to use Description Vectorstore Vectorstore No 쉽게 사용 가능 .. LangChain - 2.5 Vector stores GetStarted 참조 : https://python.langchain.com/docs/modules/data_connection/vectorstores/ Vector stores | 🦜️🔗 Langchain Head to Integrations for documentation on built-in integrations with 3rd-party vector stores. python.langchain.com Vector stores Embedding 된 vector 값을 저장하는 DB 여기서는 Chroma 기준으로 테스트 진행함 from langchain_community.document_loaders import TextLoader from langchain_openai import OpenAIEmbeddings f.. LangChain - 2.4 Text embedding models 참조 : https://python.langchain.com/docs/modules/data_connection/text_embedding/ Text embedding models | 🦜️🔗 Langchain Head to Integrations for documentation on built-in integrations with text embedding model providers. python.langchain.com Text embedding models Embedding class = text embedding models와 i/f를 위하여 design되었으며 다양한 embedding model 제공됨 (OpenAI, Hugging Face, Cohere 등) Embedding은 Text 조각의.. LangChain - 2.3 Text Splitter 참조 : https://python.langchain.com/docs/modules/data_connection/document_transformers/ Text Splitters | 🦜️🔗 Langchain Once you've loaded documents, you'll often want to transform them to better suit your application. The simplest example python.langchain.com Text Splitter 다음과 같이 동작 아주 작은 chunk 단위 (문장)로 나눔 특정 크기에 도달할때 까지 작은 단위를 큰 단위의 chunk 단위로 결합 해당 크기에 도달하면 해당 chunk를 고유한 텍스트 조작으로 만든 다음 약간 겹치는 새로운.. LangChain - 2.2 Document loaders 참조 : https://python.langchain.com/docs/modules/data_connection/document_loaders/csv CSV | 🦜️🔗 Langchain A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. python.langchain.com Document loaders Source에서 document를 load Simplest loader from langcha.. LangChain - 2.1 Retrieval concept 참조 : https://python.langchain.com/docs/modules/data_connection/ Retrieval | 🦜️🔗 Langchain Many LLM applications require user-specific data that is not part of the model's training set. python.langchain.com Retrieval LLM 애플리케이션에서 user-specific data를 필요로함 -> 이를 위해 RAG (Retrieval Augmented Generation) 제공 Concept Item Desc Etc. Document loaders 많은 다른 source로부터 document를 load함 Text Splitting Retrieva.. 이전 1 2 3 4 ··· 25 다음
