o {q ib@sjddlmZddlZddlmZmZmZmZddlm Z m Z Gddde Z dddZ Gddde Z dS)) annotationsN)AnyLiteralOptionalUnion)Language TextSplittercs2eZdZdZ  ddfd d ZdddZZS)CharacterTextSplitterz(Splitting text that looks at characters. F separatorstris_separator_regexboolkwargsrreturnNonec s"tjdi|||_||_dS)Create a new TextSplitter.N)super__init__ _separator_is_separator_regex)selfr r r __class__ri/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/langchain_text_splitters/character.pyr s zCharacterTextSplitter.__init__text list[str]cshjrjntj}t||jd}d}jo#tfdd|D}d}js.|s.j}||S)z=Split into chunks without re-inserting lookaround separators.keep_separator)z(?=z(?%s  z3CharacterTextSplitter.split_text..)rrreescape_split_text_with_regex_keep_separatorany _merge_splits)rrZ sep_patternsplitsZlookaround_prefixesZ is_lookaroundZ merge_seprr#r split_texts  z CharacterTextSplitter.split_text)r F)r r r rrrrrrr rr)__name__ __module__ __qualname____doc__rr- __classcell__rrrrr s  r rr r r$Union[bool, Literal['start', 'end']]rrcs|r]|rVtd|d||dkr#fddtdtddDnfd dtdtdD}tddkrB|d d7}|dkrNg|d ndg|}n t||}nt|}d d|DS) N()endc g|] }||dqSrr!iZ_splitsrr = z*_split_text_with_regex..rr:cr8r9rr;r=rrr>?r?cSsg|]}|dkr|qS)r%r)r!srrrr>Ls)r&splitrangelenlist)rr rr,rr=rr(4s " r(csZeZdZdZ   ddfdd Zd ddZd!ddZed"ddZe d#ddZ Z S)$RecursiveCharacterTextSplitterzSplitting text by recursively look at characters. Recursively tries to split by different characters to find one that works. NTF separatorsOptional[list[str]]rr4r rrrrrc s.tjdd|i||pgd|_||_dS)rr)r   r%Nr)rr _separatorsr)rrHrr rrrrrVs z'RecursiveCharacterTextSplitter.__init__rr rcCs(g}|d}g}t|D](\}}|jr|nt|}|dkr"|}nt||r4|}||dd}nq |jr:|nt|}t|||jd} g} |jrNdn|}| D]2} || |jkrb| | qR| rq| | |} | | g} |sy| | qR| | |} | | qR| r| | |} | | |S)z&Split incoming text and return chunks.rAr%r:Nr) enumeraterr&r'searchr(r)Z_length_function _chunk_sizeappendr+extend _split_text)rrrHZ final_chunksr Znew_separatorsr<Z_srr,Z _good_splitsrBZ merged_textZ other_inforrrrRbsD         z*RecursiveCharacterTextSplitter._split_textcCs|||jS)zSplit the input text into smaller chunks based on predefined separators. Args: text (str): The input text to be split. Returns: List[str]: A list of text chunks obtained after splitting. )rRrL)rrrrrr-s z)RecursiveCharacterTextSplitter.split_textlanguagercKs||}|d|dd|S)aReturn an instance of this class based on a specific language. This method initializes the text splitter with language-specific separators. Args: language (Language): The language to configure the text splitter for. **kwargs (Any): Additional keyword arguments to customize the splitter. Returns: RecursiveCharacterTextSplitter: An instance of the text splitter configured for the specified language. T)rHr Nr)get_separators_for_language)clsrSrrHrrr from_languages z,RecursiveCharacterTextSplitter.from_languagecCsR|tjtjfvr gdS|tjkrgdS|tjkrgdS|tjkr'gdS|tjkr0gdS|tjkr9gdS|tjkrBgdS|tj krKgdS|tj krTgd S|tj kr]gd S|tj krfgd S|tj krogd S|tjkrxgd S|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkrgdS|tjkr d}d|dd|dd|dd|dd|dd d!d"d#d$d%d&d'd(d)d*d+d,gS|tjvrd-|d.}t|d-|d/tt}t|)0a Retrieve a list of separators specific to the given language. Args: language (Language): The language for which to get the separators. Returns: List[str]: A list of separators appropriate for the specified language. ) class z void z int z float z double  if  for  while  switch  case r rJrKr%)  func  var  const  type rXrYr[r\r rJrKr%)rW public protected private  static rXrYrZr[r\r rJrKr%)rWrarbrcz internal z companion z fun  val r^rXrYrZz when r\ else r rJrKr%) function r_ let r^rWrXrYrZr[r\ default r rJrKr%) enum interface z namespace r`rWrgr_rhr^rXrYrZr[r\rir rJrKr%) rgrWrX foreach rZ do r[r\r rJrKr%) z message z service rjz option  import z syntax r rJrKr%)rW def z def r rJrKr%)z =+ z -+ z \*+ z .. * r rJrKr%) rorWrX unless rZrYrmz begin z rescue r rJrKr%)roz defp z defmodule z defprotocol z defmacro z defmacrop rXrprZr\z cond z with rYrmr rJrKr%) z fn r_rhrXrZrYz loop  match r_r rJrKr%)rWz object rorer^rXrYrZrqr\r rJrKr%)r]rW struct rjrXrYrZrmr[r\r rJrKr%) z #{1,6} z``` z \*\*\*+ z ---+ z ___+ r rJrKr%)z \\chapter{z \\section{z \\subsection{z \\subsubsection{z \\begin{enumerate}z \\begin{itemize}z \\begin{description}z \\begin{list}z \\begin{quote}z \\begin{quotation}z \\begin{verse}z \\begin{verbatim}z \\begin{align}z$$$rKr%)zs  +