o rq i'@svddlZddlmZmZmZmZddlmZddlm Z ddl m Z m Z ddl mZGdddZGd d d e ZdS) N)DictListOptionalUnion)SentencePieceProcessor)PreTrainedTokenizer) BatchEncoding EncodedInput)PaddingStrategyc @seZdZdefddZdefddZ  ddeded ed eefd d Z d eed efddZ deed efddZ ddZ ddZ dS) SPTokenizer model_pathcCstj|s J|t|d|_|j|_|j|_|j|_|j |_ |j|j ks4Jgd}i|_ i|_ |D]}|j|j |<||j |j<|jd7_q@dS)N)Z model_file)z[MASK][gMASK]z[sMASK]sopZeop)ospathisfilersp_model vocab_sizen_wordsbos_ideos_idZunk_idpad_idZget_piece_sizespecial_tokensindex_special_tokens)selfr rtokenrr/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/modelscope/models/nlp/chatglm2/tokenization.py__init__ s       zSPTokenizer.__init__scC |j|SN)rZEncodeAsPieces)rr rrrtokenize  zSPTokenizer.tokenizeFboseosreturncCs@t|tusJ|j|}|r|jg|}|r||jg}|Sr")typestrrencoderr)rr r%r&trrrr*#s   zSPTokenizer.encoder+cCr!r")rdecode)rr+rrrr,/r$zSPTokenizer.decodetokenscCs|j|}|Sr")rZ DecodePieces)rr-textrrr decode_tokens2s zSPTokenizer.decode_tokenscCs ||jvr |j|S|j|Sz2 Converts a token (str) in an id using the vocab. )rrZ PieceToIdrrrrrconvert_token_to_id6s   zSPTokenizer.convert_token_to_idcCs6||jvs||j|j|jfvs|dkrdS|j|S)=Converts an index (integer) in a token (str) using the vocab.r)rrrrrZ IdToPiecerindexrrrconvert_id_to_token<s   zSPTokenizer.convert_id_to_tokenN)FF)__name__ __module__ __qualname__r)rr#boolrintr*r,r/r2r7rrrrr s$   r c sDeZdZddiZgdZd1fdd ZddZed efd d Z ed d Z ed efddZ eddZ eddZ ddZddZddZddZdeed efddZd2d d!Zd"d#Zd2d$d%Z d2d&eed'eeed eefd(d)Zdejddfd*eeeefefd+eed,ed-eed.ee d e!f d/d0Z"Z#S)3ChatGLM2Tokenizer vocab_fileztokenizer.model)Z input_idsattention_mask position_idsleftc sJd|_||_t||_|jj|jj|jjd|_tj dd|i|dS)NZ GLMTokenizer)z padding_sider) namer>r tokenizerrrrrsuperr)rr>rDkwargs __class__rrrJs zChatGLM2Tokenizer.__init__cCs@||jvr |j|S||jjvsJ|d|j|jj|S)Nz is not a special token for )rrFrEr1rrr get_commandVs    zChatGLM2Tokenizer.get_commandr'cCdS)Nzrrrrr pad_token\zChatGLM2Tokenizer.pad_tokencC |dS)NrCrKrMrrr pad_token_id` zChatGLM2Tokenizer.pad_token_idcCrL)NzrrMrrr eos_tokendrOzChatGLM2Tokenizer.eos_tokencCrP)NrBrQrMrrr eos_token_idhrSzChatGLM2Tokenizer.eos_token_idcCs|jjSr")rFrrMrrrrlszChatGLM2Tokenizer.vocab_sizecs(fddtjD}|j|S)z Returns vocab as a dict csi|]}||qSr)_convert_id_to_token).0irMrr rs z/ChatGLM2Tokenizer.get_vocab..)rangerupdateZadded_tokens_encoder)rZvocabrrMr get_vocabps  zChatGLM2Tokenizer.get_vocabcKr!r")rFr#)rr.rHrrr _tokenizeyr$zChatGLM2Tokenizer._tokenizecCr!r0)rFr2r1rrr_convert_token_to_id| z&ChatGLM2Tokenizer._convert_token_to_idcCr!)r3)rFr7r5rrrrVr_z&ChatGLM2Tokenizer._convert_id_to_tokenr-cCr!r")rFr/)rr-rrrconvert_tokens_to_stringr$z*ChatGLM2Tokenizer.convert_tokens_to_stringNcCstj|rtj||jd}n|}t|jd }|}Wdn1s(wYt|d}||Wd|fS1sDwY|fS)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. r>rbNwb) rrisdirjoinvocab_files_namesopenr>readwrite)rZsave_directoryZfilename_prefixr>ZfinZ proto_strwriterrrrsave_vocabularys     z!ChatGLM2Tokenizer.save_vocabularycCs|d|dg}|S)Nr rrQ)r prefix_tokensrrrget_prefix_tokenssz#ChatGLM2Tokenizer.get_prefix_tokenscCsX|durg}d}t|D]\}\}}|d|d||7}q |dt|d|7}|S)Nr4u [Round {}] 问：{} 答：{} ru[Round {}] 问：{} 答：) enumerateformatlen)rqueryhistorypromptrXZ old_queryresponserrr build_prompts zChatGLM2Tokenizer.build_prompt token_ids_0 token_ids_1cCs0|}||}|dur|||dg}|S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. NrB)rlrK)rrurvrkrrr build_inputs_with_special_tokenssz2ChatGLM2Tokenizer.build_inputs_with_special_tokensencoded_inputs max_lengthpadding_strategypad_to_multiple_ofreturn_attention_maskc Cs|jdksJ||jd}t|}|tjkrt|}|dur1|dur1||dkr1||d|}|tjko;t||k}d|vrGdg||d<d|vrStt||d<|r|t|} d|vrjdg| |d|d<d|vrydg| |d|d<|jg| |||jd<|S)a? Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) rArNrr?r@) rDmodel_input_namesror ZLONGEST DO_NOT_PADlistrZrR) rrxryrzr{r|Zrequired_inputZ seq_lengthZneeds_to_be_padded differencerrr_padsN      zChatGLM2Tokenizer._pad)rAr")$r8r9r:rer}rrKpropertyr)rNrRrTrUrr\r]r^rVrr`rjrlrtr<rrwr r~rrr rr;dictr __classcell__rrrIrr=Es`          r=)rtypingrrrrZ sentencepiecerZ transformersrZ$transformers.tokenization_utils_baserr Ztransformers.utilsr r r=rrrrs   ;