o )i9@s.UddlZddlZddlZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlZddlmZmZddlmZddlmZddlmZmZmZmZmZmZdd lmZmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;ddlm?Z?e)j@rddlAmBZBnddlmBZBdZCe+eDZEde eFfddZGide-de.de1de4de7de7de0de3d e2d!e/d"e8d#e5d$e6d%e;d&e:d'e9ZHeIeFeJefeKd(<d)d*iZLeIeFeFfeKd+<d,d-id.d-id,d-id/ZMeIeFeIeFe ffeKd0<Gd1d2d2eFejNZOe d3ZP 4 4dd5e gePfd6eFd7eQd8eQdePf d9d:ZRedddd;de eFd?eeFeSdfdeTeFf d@dAZddddBde eFd=e eFd?eeFeSdfdeSf dDdEZUdFeeFefdGeFd=e eFdeSfdHdIZVdJeddfdKdLZWdMeIeFe fddfdNdOZXdJedeSfdPdQZYdJedeSfdRdSZZdJedeSfdTdUZ[dJedeSfdVd.Z\dJedeSfdWdXZ]dYeIeFe fdZeFfd[d\Z^dJedefd]d^Z_ ddFeFd_eFd`eSd=e eFde`eFeFff dadbZaddeOjbddfdFeeFefd`eSd=e eFdce eFddeOdee eIeFe fdfe e egefdefdgdhZc iddFeeFefdCeFd=e eFde efdjdkZd iddCeFdFeeFefd=e eFfdldmZeeddFeFd=e eFfdndoZfdpeFdeeFdffdqdrZge iddFeeFefd=e eFfdsdtZhddudvZi  ddFeeFefdwe eeSeFfd=e eFdeIeFe ffdxdyZjdJefdzd{Zk ddFeFd`eSd=e eFde efd|d}Zldd~dFeFd=e eFfddZm ddeeFejnfd`eSd=e eFde eIeFe ffddZodeIfddZpdeQfddZqdS)N)cachepartial)Path)AnyCallableOptionalTypeVarUnion)get_safetensors_metadatahf_hub_downloadlist_repo_files)try_to_load_from_cache)EntryNotFoundErrorHfHubHTTPErrorHFValidationErrorLocalEntryNotFoundErrorRepositoryNotFoundErrorRevisionNotFoundError)GenerationConfigPretrainedConfig)get_image_processor_config)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES)get_tokenizer_config) CONFIG_NAME)envs) init_logger) ChatGLMConfigDeepseekVLV2Config EAGLEConfig JAISConfig KimiVLConfig MedusaConfigMLPSpeculatorConfigNemotron_Nano_VL_ConfigNemotronConfig OvisConfigRWConfigSpeculatorsConfigStep3TextConfig Step3VLConfigUltravoxConfig)adapt_config_dict)check_gguf_file) AutoConfig params.jsonreturncCstd}|r |r |SdS)a) Get the HuggingFace token from environment variable. Returns None if the token is not set, is an empty string, or contains only whitespace. This follows the same pattern as huggingface_hub library which treats empty string tokens as None to avoid authentication errors. ZHF_TOKENN)osgetenvstrip)tokenr5j/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/transformers_utils/config.py _get_hf_token7s r7ZchatglmZdeepseek_vl_v2Zkimi_vlZLlama_Nemotron_Nano_VLZ RefinedWebZRefinedWebModelZjaisZmlp_speculatorZmedusaeagle speculatorsZnemotronZovisZultravoxZstep3_vlZ step3_text_CONFIG_REGISTRYZ llm_config text_config_CONFIG_ATTRS_MAPPINGZhas_no_defaults_at_initTis_encoder_decoder)Z internvl_chatZmllamaZNVLM_D_AUTO_CONFIG_KWARGS_OVERRIDESc@seZdZdZdZdZdS) ConfigFormatautoZhfZmistralN)__name__ __module__ __qualname__AUTOHFMISTRALr5r5r5r6r?lsr?_Rfunclog_msg max_retries retry_delayc Cst|D]<}z|WSty@}z(||dkr"td||td|||d|t||d9}WYd}~qd}~wwtd)Nz%s: %sz%s: %s, retrying %d of %drHzShould not be reached)range ExceptionloggererrortimesleepAssertionError)rIrJrKrLattempter5r5r6 with_retryus     rWrevision repo_typer4repo_idrYrZr4cs&dttffdd }t|dS)Nr0cstrfdddDSztjr*ddlm}|tdddWSt dWSt j j y?gYSw) Ncs"g|] }|rt|qSr5)is_filestr relative_to).0file local_pathr5r6 s  z9list_repo_files..lookup_files..*rmodelscope_list_repo_filesZMODELSCOPE_API_TOKENrYr4rX) rexistsrglobrVLLM_USE_MODELSCOPEvllm.transformers_utils.utilsrfr1r2hf_list_repo_fileshuggingface_huberrorsOfflineModeIsEnabledrer[rZrYr4rar6 lookup_filess,  z%list_repo_files..lookup_fileszError retrieving file list)listr]rW)r[rYrZr4rqr5rpr6r s r rZrYr4 file_namecCst||||d}||vS)Nrsr )r[rtrZrYr4 file_listr5r5r6 file_existss rvmodel config_namecCsNt|}r||St|||d}t|trdStt|||tdS)Nr[filenamerYTrg)rrhr\r isinstancer]rvr7)rwrxrYrbcached_filepathr5r5r6file_or_path_existss  r}configcCs@t|dd}|durt|t|dd}|durt|dSdS)z)Provide backwards compatibility for RoPE.r;N rope_scaling)getattrpatch_rope_scalingpatch_rope_scaling_dict)r~r;rr5r5r6rs   rrcCsd|vrd|vr|d}|d}||krtd|d|dd|vr2d|vr2|d|d<tdd|vr:td|ddkrKd |d<td dS|dd krbd |vsWJd |d<tddSdS)N rope_typetypez#Found conflicts between 'rope_type=z' (modern field) and 'type=z6' (legacy field). You should only specify one of them.z,Replacing legacy 'type' key with 'rope_type'z*rope_scaling should have a 'rope_type' keyZsuZlongropez/Replacing legacy rope_type 'su' with 'longrope'Zmrope mrope_sectiondefaultz1Replacing legacy rope_type 'mrope' with 'default') ValueErrorrPinfowarning)rrZrope_type_legacyr5r5r6rs,     rcCs t|dd}|dur dSd|vS)NrFrr)r~rr5r5r6 _uses_mropes rcCst|p t|p t|S)z1Detect if the model with this config uses M-ROPE.)rget_text_configthinker_uses_mroper~r5r5r6 uses_mropes rcCs8t|dd}|dur dSt|dd}|durdSt|S)zADetect if the model contains a thinker config and it uses M-ROPE.thinker_configNFr;)rr)r~rZthinker_text_configr5r5r6r s  rcCs&dtdtfdd}||p||S)zCDetect if the model with this config is used as an encoder/decoder.r~r0cSs t|ddS)Nr=Frrr5r5r6_is_encoder_decoders z/is_encoder_decoder.._is_encoder_decoder)rboolr)r~rr5r5r6r=s cCs.|}t|dd}rddh}||SdS)zR Detect if the model with this config is used with interleaved attention. layer_typesNfull_attentionsliding_attentionF)rrissubset)r~r;rZinterleaved_typesr5r5r6is_interleaved#s  rkwargs model_typecCs|tvr |t||S)zI Update kwargs for AutoConfig initialization based on model_type )r>update)rrr5r5r6 _maybe_update_auto_config_kwargs.srcCsLtD]\}}t||r#t||s||t||itd||q|S)z4Remap config attributes to match the expected names.z&Remapped config attribute '%s' to '%s')r<itemshasattrrrrPdebug)r~old_attrnew_attrr5r5r6_maybe_remap_hf_config_attrs7s  r tokenizertrust_remote_codec Kst|}|rt|j|d<t|j}nd}tj|dur|n|f||td|\}}|dd} | dur=| dd}}||fS)zS If running a speculators config, override running model with target model gguf_fileN)rYrr4speculators_configZverifierZ name_or_path)r-rnameparentrget_config_dictr7get) rwrrrYris_ggufZgguf_model_repo config_dict_Z spec_configr5r5r6,maybe_override_with_speculators_target_modelBs$   r code_revision config_formathf_overrides_kwhf_overrides_fnc Ks&t|}|rt|j|d<t|j}|tjkrMz|s!t|t|dr%tj}nt|t |dr0tj }nt dWnt yL} z dj |d} t | | d} ~ ww|tjkrtj|f||td|\} } | d} | durv| ddurtd n| } | tvrt| }|j|f||td|}n5zt|| d }tj|f|||td |}Wnt y} z|sd t| vrd }t|| | d} ~ wwt|}nf|tj krt||} | d}durt||fi|}|| d<t| }t|dd}rt|tr|jt |}||}dd|D|_!t"t#d|d|_$nddtD}t d|dd%|d|rL|j&t'vr?td|j&dt'|j&} |(d| gi| dd}|durdt|d|rdt)d||}|durl||_*|rzt+,d||(||rt+,d|||}t-||rt.|S)NrrYzCould not detect config format for no config file found. Ensure your model has either config.json (HF format) or params.json (Mistral format).aInvalid repository ID or local directory specified: '{model}'. Please verify the following requirements: 1. Provide a valid Hugging Face repository ID. 2. Specify a local directory that contains a recognized configuration file. - For Hugging Face models: ensure the presence of a 'config.json'. - For Mistral models: ensure the presence of a 'params.json'. 3. For GGUF: pass the local path of the GGUF checkpoint. Loading GGUF from a remote repo directly is not yet supported. )rw)rYrr4rrr9)r)rrYrr4z.requires you to execute the configuration filezFailed to load the model config. If the model is a custom model not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI.max_position_embeddingssliding_windowcSsg|] }|dur dndqS)Nrrr5)r_Z layer_typer5r5r6rcszget_config..cSsg|] }|tjkr|jqSr5)r?rDvalue)r_fmtr5r5r6rcszUnsupported config format: z. Supported formats are: z, ze. Ensure your model uses one of these configuration formats or specify the correct format explicitly.zCan't get gguf config for .Z architecturesquantization_configzhf_quant_config.jsonzOverriding HF config with %s)/r-rrrr?rDr}HF_CONFIG_NAMErEMISTRAL_CONFIG_NAMErFrrOformatrrr7rr:from_pretrainedrr.r] RuntimeErrorr_download_mistral_config_file_maybe_retrieve_max_pos_from_hfr,rr{rrZnum_hidden_layerslenrnextfilterrjoinrrrget_hf_file_to_dictrrPrr(maybe_register_config_serialize_by_value)rwrrYrrrrrrrV error_messagerrrZ config_classr~err_msgrrZpattern_repeatsrZsupported_formatsrr5r5r6 get_config`s                     rmaincCsVt||}|r |Szt|||d}t|trt|WSWdSty*YdSw)Nry)rr\rr{r]r)rwrtrY file_pathr|r5r5r6try_get_local_file s    rc Cst|||d}|dur[z t|||d}WnBtjjy YdSttttfy<}z t d|WYd}~dSd}~wt yV}zt j d||dWYd}~dSd}~wwt |}|dur||r|t| }t|WdS1swwYdS)a Downloads a file from the Hugging Face Hub and returns its contents as a dictionary. Parameters: - file_name (str): The name of the file to download. - model (str): The name of the model on the Hugging Face Hub. - revision (str): The specific version of the model. Returns: - config_dict (dict): A dictionary containing the contents of the downloaded file. rwrtrYNrz/File or repository not found in hf_hub_downloadzDCannot connect to Hugging Face Hub. Skipping file download for '%s':exc_info)rr rmrnrorrrrrPrrrrr\openjsonload)rtrwrYrZ hf_hub_filerVr`r5r5r6rs<   rc Csd}d}t|||drt|||}|durdStdtdd|Dd}ttdd|Dd}|r_d |d }t|||}td d|Dd}|durUt|}td ||d SdS)a This function gets the pooling and normalize config from the model - only applies to sentence-transformers models. Args: model (str): The name of the Hugging Face model. revision (str, optional): The specific version of the model to use. Defaults to 'main'. Returns: dict: A dictionary containing the pooling type and whether normalization is used. z modules.jsonN)rwrxrYz2Found sentence-transformers modules configuration.cs |] }|ddkr|VqdS)rz$sentence_transformers.models.PoolingNr5r_itemr5r5r6 h  z%get_pooling_config..csr)rz&sentence_transformers.models.NormalizeNr5rr5r5r6rlrFz{}/config.jsonpathcss |] \}}|dur|VqdS)TNr5)r_rvalr5r5r6ruszFound pooling configuration.)Z pooling_type normalize) r}rrPrrrrrget_pooling_config_name) rwrYZmodules_file_nameZ modules_dictZpoolingrZpooling_file_nameZ pooling_dictpooling_type_namer5r5r6get_pooling_configJs:     r pooling_namecCsbd|vr |dd}d|vr|dd}d|vrd}gd}|}||vr)|Std|d ) NZ pooling_mode_rrZ lasttokenlast)ZLASTALLZCLSZSTEPZMEANz Pooling type z not supported)replacesplitupperNotImplementedError)rZsupported_pooling_typesrr5r5r6rs  rcsgd}d|D]}t|||ddurt|||rnqsOt|sOz t||td}Wn ty;g}Ynw|D]}||vrNt|||rNnq>sSdStdt fdddDreSdS) a Returns the tokenization configuration dictionary for a given Sentence Transformer BERT model. Parameters: - model (str|Path): The name of the Sentence Transformer BERT model. - revision (str, optional): The revision of the m odel to use. Defaults to 'main'. Returns: - dict: A dictionary containing the configuration parameters for the Sentence Transformer BERT model. )zsentence_bert_config.jsonzsentence_roberta_config.jsonzsentence_distilbert_config.jsonzsentence_camembert_config.jsonzsentence_albert_config.jsonz sentence_xlm-roberta_config.jsonzsentence_xlnet_config.jsonNrrgz3Found sentence-transformers tokenize configuration.c3s|]}|vVqdSNr5)r_kZ encoder_dictr5r6rsz.)Zmax_seq_lengthZ do_lower_case) rrr is_absoluter r7rOrPrall)rwrYZ!sentence_transformer_config_files config_fileZ repo_filesrxr5rr6)get_sentence_transformer_tokenizer_configsH    rc szddl}d}Wn tyd}YnwzAddl}ddlddlddlm}d|ffdd }|j|||rP |dd l m }|rS|j |WdSWdSWdSt yo}zt jd |d WYd}~dSd}~ww) aTry to register HF model configuration class to serialize by value If trust_remote_code is set, and the model's config file specifies an `AutoConfig` class, then the config class is typically an instance of a custom class imported from the HF modules cache. Examples: >>> from transformers import AutoConfig >>> klass = AutoConfig.from_pretrained('meta-llama/Meta-Llama-3-8B', trust_remote_code=True) >>> klass.__class__ # transformers.models.llama.configuration_llama.LlamaConfig >>> import transformers_modules # error, not initialized >>> klass = AutoConfig.from_pretrained('deepseek-ai/DeepSeek-V2.5', trust_remote_code=True) >>> import transformers_modules # success, initialized >>> klass.__class__ # transformers_modules.deepseek-ai.DeepSeek-V2.5.98b11844770b2c3ffc18b175c758a803640f4e77.configuration_deepseek.DeepseekV2Config In the DeepSeek example, the config class is an instance of a custom class that is not serializable by default. This class will not be importable in spawned workers, and won't exist at all on other nodes, which breaks serialization of the config. In this function we tell the cloudpickle serialization library to pass instances of these generated classes by value instead of by reference, i.e. the class definition is serialized along with its data so that the class module does not need to be importable on the receiving end. See: https://github.com/cloudpipe/cloudpickle?tab=readme-ov-file#overriding-pickles-serialization-mechanism-for-importable-constructs rNTF) VllmConfigr~csj|ffSr)loadsdumpsr cloudpicklepickler5r6_reduce_configsz@maybe_register_config_serialize_by_value.._reduce_config)rayzUnable to register remote classes used by trust_remote_code with by-value serialization. This may lead to a later error. If remote code is not needed remove `--trust-remote-code`r)transformers_modules ImportErrormultiprocessingrrZ vllm.configrZreducerregisterZregister_pickle_by_valueZvllm.executor.ray_utilsrrOrPr)rZtransformers_modules_availablerrrrrVr5rr6rs6    rhf_tokencKs4tjrtSt|rt|j}t|f||d|S)N)r4rY)rrjdictr-rrr)rwrrYrr5r5r6get_hf_image_processor_configs rcCs"|}||urt|dsJ|S)zaGet the "sub" config relevant to llm for multi modal models. No op for pure text models. Znum_attention_heads)rr)r~r;r5r5r6get_hf_text_config*src CsVztj||dWSty*zt|||d}t|WYSty)YYdSww)NrrrY)rrOSErrorrZfrom_model_config)rwrrYr~r5r5r6try_get_generation_config9s"  rrcCs4tt||td}zt|dWStyYdSw)NrgzError retrieving safetensors)rr r7rWrO)rwrYZ get_safetensors_metadata_partialr5r5r6try_get_safetensors_metadataOs rpretrained_model_name_or_pathcCs&zt|||dWStyYdSw)Nr)rrO)rrrYr5r5r6try_get_tokenizer_configbs rcCs@d}t|||}|durtd|d|dt|tsJ|S)Nr/zFailed to load mistral 'z' config for model zT. Please check if the model is a mistral-format model and if the config file exists.)rrr{r)rwrYZconfig_file_namerr5r5r6rqs rc Ksvd}z|dd}t|||tjd}|j}r|}W|SW|Sty:}ztjd|dWYd}~|Sd}~ww)NirF)rwrrYrz|The params.json file is missing 'max_position_embeddings' and could not get a value from the HF config. Defaulting to 128000r) rrr?rErrrOrPr)rwrYrrZtrust_remote_code_valZ hf_configZhf_valuerVr5r5r6r}s*  r)rHrHr)r)r0N)NN)renumrr1rR functoolsrrpathlibrtypingrrrrr rmr r r rlrZhuggingface_hub.utilsrrrrrrZ transformersrrZ.transformers.models.auto.image_processing_autorZ&transformers.models.auto.modeling_autorZ*transformers.models.auto.tokenization_autorZtransformers.utilsrrZvllmrZ vllm.loggerrZvllm.transformers_utils.configsrrrr r!r"r#r$r%r&r'r(r)r*r+Z'vllm.transformers_utils.configs.mistralr,rkr-rjZ modelscoper.rrArPr]r7r:rr__annotations__r<r>Enumr?rGintrWrrrrvr}rrrrrr=rrrtuplerrDrrrrrrrrrrrPathLikerrrr5r5r5r6s           D           *          !  -   .5 >G