o ưia/@sdZddlmZddlZddlmZmZddlmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZmZdd lmZmZdd lmZdd lmZdd lmZm Z erbdd lm!Z!GdddeZ"dS)z Base RAG Ingestion class. Provides abstract methods for: - OCR - Chunking - Embedding - Vector Store operations Providers can inherit and override methods as needed. ) annotationsN)ABCabstractmethod) TYPE_CHECKINGAnyDictListOptionalTuplecast)verbose_logger)uuid4)DEFAULT_CHUNK_OVERLAPDEFAULT_CHUNK_SIZE)get_async_httpx_clienthttpxSpecialProvider)extract_text_from_pdf)RecursiveCharacterTextSplitter)RAGIngestOptionsRAGIngestResponse)Routerc@seZdZdZ d/d0ddZd1d d Zed2ddZ   d3d4ddZd5ddZ d6d!d"Z d7d%d&Z e d8d*d+Z   d3d9d-d.ZdS):BaseRAGIngestiona Base class for RAG ingestion. Providers should inherit from this class and override methods as needed. For example, OpenAI handles embedding internally when attaching files to vector stores, so it overrides the embedding step to be a no-op. Ningest_optionsrrouterOptional['Router']cCs||_||_dt|_|d|_tttt f|dp!ddi|_ |d|_ tttt f|dp6i|_ |d|_ |dS) NZingest_ocrchunking_strategytypeauto embeddingZ vector_storename)rrr ingest_idget ocr_configr rstrrrembedding_configvector_store_configZ ingest_name_load_credentials_from_config)selfrrr)[/home/app/Keep/.python/lib/python3.10/site-packages/litellm/rag/ingestion/base_ingestion.py__init__,s     zBaseRAGIngestion.__init__returnNonecCs^ddlm}|jd}|r)tjr+||}|D]\}}||jvr(||j|<qdSdSdS)z Load credentials from litellm_credential_name if provided in vector_store config. This allows users to specify a credential name in the vector_store config which will be resolved from litellm.credential_list. r)CredentialAccessorZlitellm_credential_nameN)Z.litellm.litellm_core_utils.credential_accessorr.r&r"litellmZcredential_listZget_credential_valuesitems)r(r.Zcredential_nameZcredential_valueskeyvaluer)r)r*r'Ds      z.BaseRAGIngestion._load_credentials_from_configr$cCs|jddS)zGet the vector store provider.custom_llm_providerZopenai)r&r")r(r)r)r*r3Usz$BaseRAGIngestion.custom_llm_provider file_data Optional[Tuple[str, bytes, str]]file_url Optional[str]file_idCTuple[Optional[str], Optional[bytes], Optional[str], Optional[str]]c s|r|\}}}|||dfS|r;ttjd}||IdH}||j}|ddp-d}|jdd}|||dfS|rCddd|fStd) aG Upload / prepare file for ingestion. Args: file_data: Tuple of (filename, content_bytes, content_type) file_url: URL to fetch file from file_id: Existing file ID to use Returns: Tuple of (filename, file_content, content_type, existing_file_id) N)Z llm_provider/documentz content-typezapplication/octet-streamz,Must provide file_data, file_url, or file_id) rrZRAGr"raise_for_statuscontentsplitheaders ValueError) r(r4r6r8filename file_content content_type http_clientresponser)r)r*uploadZs     zBaseRAGIngestion.uploadrCOptional[bytes]rDc s|jr|sdS|jdd}|rd|vrd\}}nd\}}t|d}d|d |}|jdurC|jj|d |||id IdH}ntj|d |||id IdH}t|d rd|j rdd dd|j DSdS)z Perform OCR on file content to extract text. Args: file_content: Raw file bytes content_type: MIME type of the file Returns: Extracted text or None if OCR not configured/needed Nmodelzmistral/mistral-ocr-latestimage) image_urlrK) document_urlrLutf-8zdata:z;base64,r)rIr<pagesz css |] }t|dr|jVqdS)markdownN)hasattrrO).0pager)r)r* s   z'BaseRAGIngestion.ocr..) r#r"base64 b64encodedecoderZaocrr/rPrNjoin) r(rCrDZ ocr_modelZdoc_typeZurl_keyZ b64_contentZdata_urlZ ocr_responser)r)r*r}s.        zBaseRAGIngestion.ocrtext ocr_was_usedbool List[str]c Csd}|r|}n8|r?|s?z|d}Wn,ty>|dr3tdt|}|s2tdgYSn tdgYSYnw|sCgS|jpGi}|dt}|dt }|d d}||d } |re|| d <t d i| } | |S) a Split text into chunks using RecursiveCharacterTextSplitter. Args: text: Text from OCR (if used) file_content: Raw file content bytes ocr_was_used: Whether OCR was performed Returns: List of text chunks NrMs%PDFz(PDF detected, attempting text extractionzkPDF text extraction failed. Install 'pypdf' or 'PyPDF2' for PDF support, or enable OCR with a vision model.z,Binary file detected, skipping text chunking chunk_size chunk_overlap separators)r\r]r)) rVUnicodeDecodeError startswithr debugrrr"rrrZ split_text) r(rXrCrYZ text_to_chunkZ splitter_argsr\r]r^Zsplitter_kwargsZ text_splitterr)r)r*chunksB         zBaseRAGIngestion.chunkchunksOptional[List[List[float]]]csd|jr|sdS|jdd}|jdur |jj||dIdH}n tj||dIdH}dd|jDS)z Generate embeddings for text chunks. Args: chunks: List of text chunks Returns: List of embeddings or None NrIztext-embedding-3-small)rIinputcSsg|]}|dqS)rr))rQitemr)r)r* sz*BaseRAGIngestion.embed..)r%r"rZ aembeddingr/data)r(rcZembedding_modelrFr)r)r*embeds  zBaseRAGIngestion.embedrB embeddings#Tuple[Optional[str], Optional[str]]csdS)a Store content in vector store. This method must be implemented by provider-specific subclasses. Args: file_content: Raw file bytes filename: Name of the file content_type: MIME type chunks: Text chunks (if chunking was done locally) embeddings: Embeddings (if embedding was done locally) Returns: Tuple of (vector_store_id, file_id) Nr))r(rCrBrDrcrjr)r)r*storeszBaseRAGIngestion.storerc s|j|||dIdH\}}}}z;|j||dIdH}|j|||jdud} |j| dIdH} |j|||| | dIdH\} } t|jd| pEd| pH|d WStyq} zt d | t|jd ddt | d WYd} ~ Sd} ~ ww) as Execute the full ingestion pipeline. Args: file_data: Tuple of (filename, content_bytes, content_type) file_url: URL to fetch file from file_id: Existing file ID to use Returns: RAGIngestResponse with status and IDs Raises: ValueError: If no input source is provided )r4r6r8N)rCrD)rXrCrY)rc)rCrBrDrcrj completed)idstatusvector_store_idr8zRAG Pipeline failed: failed)rorprqr8error) rGrrbr#rirlrr! Exceptionr exceptionr$)r(r4r6r8rBrCrDZexisting_file_idZextracted_textrcrjrqZresult_file_ider)r)r*ingestsR zBaseRAGIngestion.ingest)N)rrrr)r,r-)r,r$)NNN)r4r5r6r7r8r7r,r9)rCrHrDr7r,r7)rXr7rCrHrYrZr,r[)rcr[r,rd) rCrHrBr7rDr7rcr[rjrdr,rk)r4r5r6r7r8r7r,r)__name__ __module__ __qualname____doc__r+r'propertyr3rGrrbrirrlrwr)r)r)r*r#s(   # 2 < r)#r{ __future__rrTabcrrtypingrrrrr r r r/Zlitellm._loggingr Z litellm._uuidr Zlitellm.constantsrrZ&litellm.llms.custom_httpx.http_handlerrrZ"litellm.rag.ingestion.file_parsersrZlitellm.rag.text_splittersrZlitellm.types.ragrrrrr)r)r)r*s  $