o )i,@sddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z mZddlZddlmZmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZe rudd l m!Z!ddl"m#Z#ddl$m%Z%ne Z!e Z#e Z%ee&Z'eeee%fZ(ddde(de)e*de e+de,fddZ-ddddde(de,de e+de e*de e+de)e*f ddZ.de(de(fddZ/deddfd d!Z0d"d#ddd$d%ee,efd&e,d'e+d(e e,d)e e,de(f d*d+Z1ee1Z2d,e!d-e fd.d/Z3d0e#de e(fd1d2Z4ee4Z5dS)3N) lru_cache)Path) MethodType) TYPE_CHECKINGAnyOptionalUnion) AutoTokenizerPreTrainedTokenizerPreTrainedTokenizerFast)envs) init_logger))get_sentence_transformer_tokenizer_config)MistralTokenizer)check_gguf_file) make_async) ModelConfig) LoRARequest) TokenizerBaseskip_special_tokens tokenizer token_idsrreturncCs*t|d|j}|dur|||dS||S)z Backend-agnostic equivalent of HF's `tokenizer.decode(token_ids, ...)`. `skip_special_tokens=None` means to use the backend's default settings. _decodeNr)getattrdecode)rrrZ decode_methodrm/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/transformers_utils/tokenizer.py decode_tokens(s  r) truncation max_lengthadd_special_tokenstextr r!r"cCsFi}|dur ||d<|dur||d<|dur||d<|j|fi|S)z Backend-agnostic equivalent of HF's `tokenizer.encode(text, ...)`. `add_special_tokens=None` means to use the backend's default settings. Nr!r r")encode)rr#r r!r"Zkw_argsrrr encode_tokens=sr%cst}jjjtttdr.CachedTokenizerrcSNrself)tokenizer_all_special_idsrrall_special_idstz=get_cached_tokenizer..CachedTokenizer.all_special_idscr'r(rr))tokenizer_all_special_tokensrrall_special_tokensxr-z@get_cached_tokenizer..CachedTokenizer.all_special_tokenscr'r(rr))%tokenizer_all_special_tokens_extendedrrall_special_tokens_extended|r-zIget_cached_tokenizer..CachedTokenizer.all_special_tokens_extendedcr'r(rr)) max_token_idrrr2r-z:get_cached_tokenizer..CachedTokenizer.max_token_idcr'r(rr))tokenizer_vocabrr get_vocabz7get_cached_tokenizer..CachedTokenizer.get_vocabcr'r(rr)) tokenizer_lenrr__len__r5z5get_cached_tokenizer..CachedTokenizer.__len__cs tffSr()get_cached_tokenizerr))rrr __reduce__s z8get_cached_tokenizer..CachedTokenizer.__reduce__N)__name__ __module__ __qualname__propertylistintr,strr/r1r2dictr4r7r9rr2rr+r.r0r6r3rrCachedTokenizerrsrCZCached)copyr,r/r1r4lenmaxvalueshasattr contextlibsuppressNotImplementedErrorr& __class__r:)rZcached_tokenizerrCrrBrr8Zs    "r8cs8jdddtdttffdd}t|_dS)z@Patch _pad method to accept `padding_side` for older tokenizers.N) padding_sider*rMcs@|dur||jkrdtjd}tj|dd|i|S)Nz,`padding_side` argument is not supported by z and will be ignored. stacklevel)rMtyper:warningswarn)r*rMargskwargsmsgZorig_padrrr_pads z patch_padding_side.._pad)rXr rr@r)rrXrrWrpatch_padding_sides rYautoF)tokenizer_modetrust_remote_coderevision download_dirtokenizer_namer[r\r]r^c OsFtjr9ddlm}ddlm}tj|s9|||||||t j j gdd} | }Wdn1s4wY|dkrK| dd rGt d d |d<d |vrSd |d <t|} | ret|j|d <t|j}t|dddk} | r~|dkr~tjdtdd|dkrtjt||d} | S|dkrddlm} | jt|g|R||d|} | Sztj|g|R||d|} Wn#t y}z|sdt|vsdt|vrd}t|||d}~wwt||}t|t r| dd rdd| j!"D}| #|t$| j%dvrt| t&sJt'| t| t(st)*d t+| } | S)!zMGets a tokenizer for the given model name via HuggingFace or ModelScope. r)snapshot_download)get_lock)z.*.ptz.*.safetensorsz.*.bin)Zmodel_id cache_dirr]Zlocal_files_onlyZignore_file_patternNZslowZuse_fastFz5Cannot use the fast tokenizer in slow tokenizer mode.Ztruncation_sideleftZ gguf_file/Z mistralaiZmistralz{It is strongly recommended to run mistral models with `--tokenizer-mode "mistral"` to ensure correct encoding and decoding.rNrO)r]Zcustom)TokenizerRegistry)r]r^)r\r]z,does not exist or is not currently imported.z*requires you to execute the tokenizer filezFailed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI.Z do_lower_casecSsi|] \}}||qSr)lower).0kvrrr sz!get_tokenizer..)ZChatGLMTokenizerZChatGLM4TokenizerziUsing a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.),r ZVLLM_USE_MODELSCOPEZ modelscope.hub.snapshot_downloadr`Z-vllm.model_executor.model_loader.weight_utilsraospathexistshuggingface_hub constantsZHF_HUB_OFFLINEget ValueErrorrrnameparentr@splitrRrS FutureWarningrZfrom_pretrained&vllm.transformers_utils.tokenizer_basere get_tokenizerr RuntimeErrorr isinstancerAspecial_tokens_mapitemsr"rQr:r rYr loggerwarningr8)r_r[r\r]r^rTrUr`raZtokenizer_pathZis_ggufZis_from_mistral_orgrreeerr_msgZencoder_configrzrrrrws        :  6      rw model_configrUcKs t|jf|j|j|jd|S)N)r[r]r\)cached_get_tokenizerrr[Ztokenizer_revisionr\)rrUrrrcached_tokenizer_from_config$sr lora_requestc Osf|durdSzt|jg|Ri|}W|Sty2}ztd|j|d}WYd}~|Sd}~ww)NzMNo tokenizer found in %s, using base model tokenizer instead. (Exception: %s))rwZ lora_path Exceptionr|r})rrTrUrr~rrrget_lora_tokenizer1sr)6rIrDrkrR functoolsrpathlibrtypesrtypingrrrrrnZ transformersr r r Zvllmr Z vllm.loggerr Zvllm.transformers_utils.configrZ"vllm.transformers_utils.tokenizersrZvllm.transformers_utils.utilsrZ vllm.utilsrZ vllm.configrZvllm.lora.requestrrvrr:r|Z AnyTokenizerr>r?boolr@rr%r8rYrwrrrZget_lora_tokenizer_asyncrrrrs             9  z