o `+ i2@sddlmZddlZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZddlmZmZddlmZe rTdd lmZmZmZdd lmZeeZ ed d d Z!Gdd d eeZ"Gddde"Z#Gddde$e Z%eddGdddZ&dddZ'dS)) annotationsN)ABCabstractmethod) dataclass)Enum) TYPE_CHECKINGAnyCallableLiteralOptionalTypeVarUnion)BaseDocumentTransformerDocument)Self) CollectionIterableSequence)SetTS TextSplitter)boundc@seZdZdZddedddfd@ddZedAddZ dBdCddZdDd"d#Z dEd'd(Z dFd+d,Z e dGd0d1Z e d2ded3fdHd;d<ZdId>d?ZdS)Jrz)Interface for splitting text into chunks.iFT chunk_sizeint chunk_overlaplength_functionCallable[[str], int]keep_separator$Union[bool, Literal['start', 'end']]add_start_indexboolstrip_whitespacereturnNonecCs~|dkr d|}t||dkrd|}t|||kr+d|d|d}t|||_||_||_||_||_||_dS)adCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_functionZ_keep_separator_add_start_index_strip_whitespace)selfrrrrr r"msgr-i/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/langchain_text_splitters/base.py__init__!s$   zTextSplitter.__init__textstr list[str]cCsdS)z$Split text into multiple components.Nr-)r+r0r-r-r. split_textIszTextSplitter.split_textNtexts metadatasOptional[list[dict[Any, Any]]]list[Document]c Cs|pigt|}g}t|D]=\}}d}d}||D]/} t||} |jr@|||j} || td| }|| d<t| }t | | d} | | qq|S)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater3copydeepcopyr)r'findmaxrappend) r+r4r5Z _metadatas documentsir0indexZprevious_chunk_lenchunkr:offsetZnew_docr-r-r.create_documentsMs    zTextSplitter.create_documentsrBIterable[Document]cCs:gg}}|D]}||j||jq|j||dS)zSplit documents.)r5)rAr9r:rG)r+rBr4r5docr-r-r.split_documentsas  zTextSplitter.split_documentsdocs separator Optional[str]cCs(||}|jr |}|dkrdS|S)N)joinr*strip)r+rKrLr0r-r-r. _join_docsis zTextSplitter._join_docssplits Iterable[str]c Csz||}g}g}d}|D]}||}||t|dkr|nd|jkr||jkr6td|d|jt|dkr|||} | durK|| ||jkse||t|dkr[|nd|jkr|dkr|||dt|dkrt|nd8}|dd}||jkse||t|dkr|nd|jkr|dkse||||t|dkr|nd7}q |||} | dur|| |S)NrzCreated a chunk of size z%, which is longer than the specified )r(r;r&loggerwarningrQrAr') r+rRrLZ separator_lenrKZ current_doctotald_lenrIr-r-r. _merge_splitsqsN            zTextSplitter._merge_splits tokenizerrkwargsc snzddlm}t|sd}t|d fdd }Wnty-}zd }t||d }~ww|dd |i|S)z>Text splitter that uses HuggingFace tokenizer to count length.r)PreTrainedTokenizerBasezATokenizer received was not an instance of PreTrainedTokenizerBaser0r1r#rcst|SN)r;tokenizer0r[r-r._huggingface_tokenizer_lengthszNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengthz`Could not import transformers python package. Please install it with `pip install transformers`.Nrr0r1r#rr-)Z$transformers.tokenization_utils_baser] isinstancer% ImportError)clsr[r\r]r,rberrr-rar.from_huggingface_tokenizers   z'TextSplitter.from_huggingface_tokenizergpt2all encoding_name model_nameallowed_special'Union[Literal['all'], AbstractSet[str]]disallowed_special&Union[Literal['all'], Collection[str]]rc szddl}Wnty}zd}t||d}~ww|dur$||n||d fdd } t|trD||d } i|| }|d d | i|S)z9Text splitter that uses tiktoken encoder to count length.rNzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.r0r1r#rcstj|dSN)rmro)r;encoder`rmroencr-r._tiktoken_encodersz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoder)rkrlrmrorrcr-)tiktokenreencoding_for_model get_encoding issubclassTokenTextSplitter) rfrkrlrmror\rvrgr,ru extra_kwargsr-rsr.from_tiktoken_encoders(      z"TextSplitter.from_tiktoken_encoderSequence[Document]cKs|t|S)z2Transform sequence of documents by splitting them.)rJlist)r+rBr\r-r-r.transform_documentssz TextSplitter.transform_documents)rrrrrrrrr r!r"r!r#r$r0r1r#r2r^)r4r2r5r6r#r7)rBrHr#r7)rKr2rLr1r#rM)rRrSrLr1r#r2)r[rr\rr#r) rkr1rlrMrmrnrorpr\rr#r)rBr}r\rr#r})__name__ __module__ __qualname____doc__r;r/rr3rGrJrQrZ classmethodrhsetr|rr-r-r-r.rs2 (    * ,cs8eZdZdZddedfdfdd ZdddZZS)rzz/Splitting text to tokens using model tokenizer.riNrjrkr1rlrMrmrnrorpr\rr#r$c sztjdi|zddl}Wnty"}zd}t||d}~ww|dur-||} n||} | |_||_||_dS)zCreate a new TextSplitter.rNzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.r-) superr/rvrerwrx _tokenizer_allowed_special_disallowed_special) r+rkrlrmror\rvrgr,rt __class__r-r.r/s      zTokenTextSplitter.__init__r0r2cs2d fdd }tjjjj|d}t||dS) aSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text (str): The input text to be split into smaller chunks. Returns: List[str]: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. _textr1r# list[int]csjj|jjdSrq)rrrrr)rr+r-r._encodes z-TokenTextSplitter.split_text.._encode)rtokens_per_chunkdecoderr)r0r[N)rr1r#r) Tokenizerr'r&rrsplit_text_on_tokens)r+r0rr[r-rr.r3s zTokenTextSplitter.split_text) rkr1rlrMrmrnrorpr\rr#r$r)rrrrrr/r3 __classcell__r-r-rr.rzsrzc@s|eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdS)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlZsolcsharpcobolcluaperlhaskellelixir powershellZ visualbasic6N)rrrrZCPPZGOZJAVAZKOTLINZJSrPHPPROTOPYTHONZRSTZRUBYZRUSTZSCALAZSWIFTMARKDOWNZLATEXHTMLZSOLZCSHARPCOBOLCZLUAZPERLZHASKELLZELIXIRZ POWERSHELLZ VISUALBASIC6r-r-r-r.r&s:rT)frozenc@s8eZdZUdZded< ded< ded< ded<d S) rzTokenizer data class.rrrzCallable[[list[int]], str]rzCallable[[str], list[int]]rrN)rrrr__annotations__r-r-r-r.rFs rr0r1r[r#r2cCsg}||}d}t||jt|}|||}|t|krN||||t|kr0 |S||j|j7}t||jt|}|||}|t|ks|S)z6Split incoming text and return chunks using tokenizer.r)rrminrr;rArr)r0r[rRZ input_idsZ start_idxZcur_idxZ chunk_idsr-r-r.rTs      r)r0r1r[rr#r2)( __future__rr=loggingabcrr dataclassesrenumrtypingrrr r r r r Zlangchain_core.documentsrrZtyping_extensionsrcollections.abcrrrr AbstractSet getLoggerrrUrrrzr1rrrr-r-r-r.s*   $    I@