o )i&- @sddlmZddlmZddlmZddlmZddlm Z m Z ddl m Z ddl mZddlmZmZmZmZmZmZmZdd lmZdd lmZdd lmZmZdd lmZee Z!eGd ddZ"eGdddZ#GdddZ$dej%fde&de&dedefddZ'dS))Lock) defaultdict) dataclass) HTTPStatus)OptionalUnion) ModelConfig) EngineClient) ErrorInfo ErrorResponseLoadLoRAAdapterRequest ModelCard ModelListModelPermissionUnloadLoRAAdapterRequest) init_logger) LoRARequest) LoRAResolverLoRAResolverRegistry) AtomicCounterc@seZdZUeed<eed<dS) BaseModelPathname model_pathN)__name__ __module__ __qualname__str__annotations__rrr/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_models.pyrs  rc@s.eZdZUeed<eed<dZeeed<dS)LoRAModulePathrpathNbase_model_name)rrrrrr"rrrrrr s r c seZdZdZdddededeedeee ffdd Z d d Z d e fd dZ d!deed efddZd efddZ d!dedeed eeeffddZded eeeffddZded eefddZded eefddZded eeeffdd ZZS)"OpenAIServingModelszShared instance to hold data about the loaded base model(s) and adapters. Handles the routes: - /v1/models - /v1/load_lora_adapter - /v1/unload_lora_adapter N) lora_modules engine_client model_configbase_model_pathsr$csnt||_|j|_||_||_||_i|_td|_ g|_ t D] }|j t |q$tt|_dS)Nr)super__init__r' max_model_lenr%r&static_lora_modules lora_requestsrlora_id_counterlora_resolversrZget_supported_resolversappendZ get_resolverrrlora_resolver_lock)selfr%r&r'r$Zlora_resolver_name __class__rrr)/s   zOpenAIServingModels.__init__cs\|jdurdS|jD] }t|j|jd}|j||jdIdH}t|tr+t|j j q dS)zALoads all static LoRA modules. Raises if any fail to loadN) lora_path lora_name)requestr") r+r r!rload_lora_adapterr" isinstancer ValueErrorerrormessage)r1loraZ load_requestZ load_resultrrrinit_static_lorasJs    z%OpenAIServingModels.init_static_lorasreturncstfdd|jDS)Nc3s|]}|jkVqdSN)r).0model model_namerr Xsz4OpenAIServingModels.is_base_model..)anyr')r1rCrrBr is_base_modelWsz!OpenAIServingModels.is_base_model lora_requestcCs|dur|jS|jdjS)a&Returns the appropriate model name depending on the availability and support of the LoRA or base model. Parameters: - lora: LoRARequest that contain a base_model_name. Returns: - str: The name of the base model or the first available model path. Nr)r5r'r)r1rGrrrrCZs zOpenAIServingModels.model_namecsBfddjD}fddjD}||t|dS)zMShow available models. This includes the base model and all adapterscs&g|]}t|jj|jtgdqS))idr*root permission)r rr*rr)r@Z base_modelr1rr isz=OpenAIServingModels.show_available_models..cs8g|]}t|j|j|jr|jnjdjtgdqS)r)rHrIparentrJ)r r5Z local_pathr"r'rr)r@r<rKrrrLps  )data)r'r,valuesextendr)r1Z model_cardsZ lora_cardsrrKrshow_available_modelsfs    z)OpenAIServingModels.show_available_modelsr6r"c sX|j}|j|4IdH||IdH}|dur&|WdIdHS|j}|jd}t|||d}|durB||rB||_z |j |IdHWn5t y}z)d} t j } dt|vred} t j} tt|| | dWYd}~WdIdHSd}~ww||j|<td||d|d WdIdHS1IdHswYdS) N)r5 lora_int_idr4BadRequestErrorzNo adapter found NotFoundErrorr;err_type status_codez-Loaded new LoRA adapter: name '%s', path '%s'Success: LoRA adapter 'z' added successfully.)r5r0 _check_load_lora_adapter_requestr4r-incrrFr"r%add_lora Exceptionr BAD_REQUESTr NOT_FOUNDcreate_error_responser,loggerinfo) r1r6r"r5error_check_retr4 unique_idrGeZ error_typerXrrrr7{sP      0z%OpenAIServingModels.load_lora_adapterc s|j}|j|4IdH3||IdH}|dur&|WdIdHS|j|=td|d|dWdIdHS1IdHsFwYdS)NzRemoved LoRA adapter: name '%s'rYz' removed successfully.)r5r0"_check_unload_lora_adapter_requestr,rarb)r1r6r5rcrrrunload_lora_adapters   0z'OpenAIServingModels.unload_lora_adaptercsH|jr|jstddtjdS|j|jvr"td|jddtjdSdS)Nz2Both 'lora_name' and 'lora_path' must be provided.InvalidUserInputrVThe lora adapter 'z' has already been loaded.)r5r4r`rr^r,r1r6rrrrZs   z4OpenAIServingModels._check_load_lora_adapter_requestcsB|js tddtjdS|j|jvrtd|jddtjdSdS)Nz:'lora_name' needs to be provided to unload a LoRA adapter.rhrVriz' cannot be found.rU)r5r`rr^r,r_rjrrrrfs  z6OpenAIServingModels._check_unload_lora_adapter_requestr5c s|j|4IdH||jvr|j|WdIdHS|jj}|jd}d}|jD]W}|||IdH}|durd}||_z&|j |IdH||j|<t d||j j|WWdIdHSty}zt d||j j|WYd}~q.d}~wwq.|rtd|dd tjd WdIdHStd |d d tjd WdIdHS1IdHswYdS)adAttempt to resolve a LoRA adapter using available resolvers. Args: lora_name: Name/identifier of the LoRA adapter Returns: LoRARequest if found and loaded successfully. ErrorResponse (404) if no resolver finds the adapter. ErrorResponse (400) if adapter(s) are found but none load. NrRFTz.Resolved and loaded LoRA adapter '%s' using %szBFailed to load LoRA '%s' resolved by %s: %s. Trying next resolver.zLoRA adapter 'z$' was found but could not be loaded.rTrVz LoRA adapter z does not existrU)r0r,r&rAr-r[r. resolve_lorarSr%r\rarbr3r BaseExceptionwarningr`rr^r_)r1r5r"rdZ found_adapterresolverrGrerrrrks^       ) 0z OpenAIServingModels.resolve_lorar?)rrr__doc__r rlistrrr r)r=boolrFrrrCrrQr rr r7rrgrZrfrk __classcell__rrr2rr#&sX     +     r#rTr;rWrXr>cCstt|||jddS)N)r;typecode)r:)r r valuerVrrrr`s r`N)(asyncior collectionsr dataclassesrhttprtypingrrZ vllm.configrZvllm.engine.protocolr Z vllm.entrypoints.openai.protocolr r r r rrrZ vllm.loggerrZvllm.lora.requestrZvllm.lora.resolverrrZ vllm.utilsrrrarr r#r^rr`rrrrs<      $   x