o )il@sddlZddlZddlmZmZmZddlmZddlmZm Z m Z ddl Z ddl mZddlmZmZddlmZddlmZddl mZdd lmZdd lmZdd lmZdd lmZdd lm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:ddl;mZ>m?Z?ddl@mAZAmBZBddlCmDZDddlEmFZFddlGmHZHddlImJZJmKZKddlLmMZMdd lNmOZOeePZQGd!d"d"eZRdS)#N)AsyncGeneratorIterableMapping)copy)AnyOptionalUnion) ModelConfig VllmConfig)AsyncEngineArgs) EngineClient)VLLM_V1_OUTPUT_PROC_CHUNK_SIZE) PromptType)InputPreprocessor) init_logger) LoRARequest)MULTIMODAL_REGISTRYMultiModalRegistry)PoolingRequestOutput RequestOutput) PoolingParams)SamplingParams) SupportedTask)(maybe_register_config_serialize_by_value) AnyTokenizer)init_tokenizer_from_configs) UsageContext)Deviceas_listcancel_task_threadsafecdivdeprecate_kwargs)EngineCoreRequest)EngineCoreClient)EngineDeadErrorEngineGenerateError)OutputProcessorRequestOutputCollector) ParentRequest) Processor)Executor)StatLoggerFactoryStatLoggerManager)shutdown_prometheus)IterationStatsc@sneZdZejedddddddf dedeede d ed e d e d e d e de e e de eeefdededdfddZeeddddejdddddddf ded e d ede e e de de de eeefdedede ddfddZedejdfded e d ede e e ddf ddZdd Zd!d"Zdeed#ffd$d%Z      dd&ed'ed(eeefd)e e d*e e!d+e eee"fd,e e#eefd-ed.e ede$fd/d0Z%d1e&d'e ed2e e'd3ed4e$f d5d6Z(    dd'ed7ed&ed*e e!d,e e#eefd-ed.e ede)e*dffd8d9Z+d:d;Z,d&eee-efddfded&ed*e e!d,e e#eefd-ed+e eee"fde)e/dffd?d@Z0defdAdBZ1de2fdCdDZ3dEdFZ4de5fdGdHZ6 dd*e e!de7fdIdJZ8de fdKdLZ9  d ddMdNZ:ddOdPZ;ddQdRZ ddWe e?ddfdXdYZ@ddZeddfd[d\ZAdd]e e eddfd^d_ZBde fd`daZCd*e!de fdbdcZDddede fdedfZEdeFefdgdhZGddede fdidjZH  k ddledme e dnedoe efdpdqZIddsefdtduZJ rddvedsefdwdxZKeLde fdydzZMeLde fd{d|ZNeLde fd}d~ZOeLdePfddZQdS)AsyncLLMFTNr vllm_configexecutor_class log_stats usage_context mm_registryuse_cached_outputs log_requestsstart_engine_loop stat_loggersclient_addresses client_count client_indexreturnc Cstjstdt|j|_||_||_||_|jjrd|_ n t |j|j |j d|_ t ||j |d|_t|j |jd|_tj|||j| | | d|_d|_|jr^t||jj| d|_|jd|_z t|WdStyvYdSw)a Create an AsyncLLM. Args: vllm_config: global configuration. executor_class: an Executor impl, e.g. MultiprocExecutor. log_stats: Whether to log stats. usage_context: Usage context of the LLM. mm_registry: Multi-modal registry. use_cached_outputs: Whether to use cached outputs. log_requests: Whether to log requests. start_engine_loop: Whether to start the engine loop. stat_loggers: customized stat loggers for the engine. If not provided, default stat loggers will be used. PLEASE BE AWARE THAT STAT LOGGER IS NOT STABLE IN V1, AND ITS BASE CLASS INTERFACE MIGHT CHANGE. Returns: None Using V1 AsyncLLMEngine, but envs.VLLM_USE_V1=False. This should not happen. As a workaround, try using AsyncLLMEngine.from_vllm_config(...) or explicitly set VLLM_USE_V1=0 or 1 and report this issue on Github.N) model_configscheduler_config lora_config)r1 tokenizerr5)r3)r1r2r3r:r;r<r1Z engine_idxsZcustom_stat_loggers)envs VLLM_USE_V1 ValueErrorrr?r1r7r3Zskip_tokenizer_initrBrr@rAr) processorr&output_processorr#Zmake_async_mp_client engine_corelogger_managerr,Zengine_ranks_managedZlog_engine_initializedoutput_handlerasyncioget_running_loop_run_output_handler RuntimeError) selfr1r2r3r4r5r6r7r8r9r:r;r<rQd/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py__init__1s^#   zAsyncLLM.__init__disable_log_requestszEThis argument will have no effect. Use `enable_log_requests` instead.)Zadditional_messageenable_log_requestsdisable_log_statsc Cs2tjstd||t||||| |||| d S)Nr>) r1r2r8r9r7r3r4r:r;r<)rDrErFr* get_class) clsr1r8r4r9rUrVr:r;r<rTrQrQrRfrom_vllm_configs zAsyncLLM.from_vllm_config engine_argsc Cs0||}t|}||||j|j |||dS)z'Create an AsyncLLM from the EngineArgs.)r1r2r7r3r8r4r9)Zcreate_engine_configr*rWrUrV)rXrZr8r4r9r1r2rQrQrRfrom_engine_argss zAsyncLLM.from_engine_argscCs |dSN)shutdownrPrQrQrR__del__s zAsyncLLM.__del__cCs2tt|dd}r|tt|dddS)z2Shutdown, cleaning up the background proc and IPC.rINrK)r-getattrr]r)rPrIrQrQrRr]szAsyncLLM.shutdown.c|jIdHSr\)rIZget_supported_tasks_asyncr^rQrQrRget_supported_taskszAsyncLLM.get_supported_tasks request_idpromptparams arrival_time lora_requesttokenization_kwargs trace_headersprioritydata_parallel_rankc  s|jrtt|t} t|jd} |j||||||||| \} } | s)|jdkr7| | | dd| IdH| St ||}t |jD](}| |\}}||jdkrS| nt | }||_||_| || ||| IdHqA| S)z Add new request to the AsyncLLM.) output_kindr0Nr)erroredr$ isinstancerr'rmrGZprocess_inputsn _add_requestr(rangeZget_child_inforrdsampling_params)rPrdrerfrgrhrirjrkrlZ is_poolingqueueZ prompt_strrequestZparent_requestidxZ child_requestrQrQrR add_requests,      zAsyncLLM.add_requestru parent_reqindexrtcsD|j||||||j|IdH|jr td|jdSdS)NzAdded request %s.)rHrwrIZadd_request_asyncr7loggerinford)rPrurerxryrtrQrQrRrqszAsyncLLM._add_requestrsc Csz0||j|||||||dIdH}d} | s/|p#|IdH} | j} | V| rWdSWdStjtfyM||IdH|j rLt d|t y]|j r\t d|t ym|j rlt d|ty} z||IdH|j rt d|t| d} ~ ww)aj Main function called by the API server to kick off a request * 1) Making an AsyncStream corresponding to the Request. * 2) Processing the Input. * 3) Adding the Request to the Detokenizer. * 4) Adding the Request to the EngineCore (separate process). A separate output_handler loop runs in a background AsyncIO task, pulling outputs from EngineCore and putting them into the per-request AsyncStream. The caller of generate() iterates the returned AsyncGenerator, returning the RequestOutput back to the caller. )rhrjrkrlNFRequest %s aborted. Request %s failed (engine dead). Request %s failed (bad request).Request %s failed.)rNrw get_nowaitgetfinishedrLCancelledError GeneratorExitabortr7rzr{r$rF Exceptionr%) rPrersrdrhrjrkrlqrouterQrQrRgenerate"sL         zAsyncLLM.generatecsJ|jdurdS|j|j|j|jfdd}t||_dS)zBBackground loop: pulls from EngineCore and pushes to AsyncStreams.Nc sze IdH}t|j}r|rtnd}|tkr!|jf}n t|jt|t}t|D]*\}} ||j |}|j r@J|dt|krPt dIdH|jIdHq/rfj|j|j|dqty}ztd|WYd}~dSd}~ww)NTr0r)Z engine_idxscheduler_statsiteration_statszAsyncLLM output_handler failed.)Zget_output_asynclenoutputsr.r npZ array_splitr enumerateZprocess_outputs timestampZrequest_outputsrLsleepabort_requests_asyncZ reqs_to_abortrecordZ engine_indexrrrz exceptionZpropagate_error)rZ num_outputsrZslicesiZ outputs_sliceZprocessed_outputsrrIr3rJrHrQrRrKsN    * z4AsyncLLM._run_output_handler..output_handler)rKrIrHr3rJrL create_task)rPrKrQrrRrNts 0zAsyncLLM._run_output_handlercsXt|tr |fnt|}|j|}|j|IdH|jr*t dd |dSdS)z2Abort RequestId in OutputProcessor and EngineCore.NzAborted request(s) %s.,) rostrrrHZabort_requestsrIrr7rzr{join)rPrdZ request_idsZall_request_idsrQrQrRrs  zAsyncLLM.abortpooling_paramsc Cs&z7||j|||||||dIdH}d} | s6|p#|IdH} t| ts+J| j} | V| rWdSWdStjyR| |IdH|j rQt d|t yb|j rat d|tyr|j rqt d|ty} z| |IdH|j rt d|t| d} ~ ww)a2 Main function called by the API server to kick off a request * 1) Making an AsyncStream corresponding to the Request. * 2) Processing the Input. * 3) Adding the Request to the EngineCore (separate process). A separate output_handler loop runs in a background AsyncIO task, pulling outputs from EngineCore and putting them into the per-request AsyncStream. The caller of generate() iterates the returned AsyncGenerator, returning the RequestOutput back to the caller. )rhrjrkriNFr|r}r~r)rNrwrrrorrrLrrr7rzr{r$rFrr%) rPrerrdrhrjrkrirrrrrQrQrRencodesN         zAsyncLLM.encodec|jSr\)r1r^rQrQrRget_vllm_configzAsyncLLM.get_vllm_configcrr\)r?r^rQrQrRget_model_configrzAsyncLLM.get_model_configcs td)NzNot Supported on V1 yet.)rFr^rQrQrRget_decoding_configzAsyncLLM.get_decoding_configcs |jjSr\)rGZinput_preprocessorr^rQrQrRget_input_preprocessorrzAsyncLLM.get_input_preprocessorcs |jdur td|j|S)Nz;Unable to get tokenizer because skip_tokenizer_init is True)rBrFZget_lora_tokenizerrPrhrQrQrR get_tokenizers  zAsyncLLM.get_tokenizercsdSNFrQr^rQrQrRis_tracing_enabled$szAsyncLLM.is_tracing_enabledcs|jr |jdSdSr\)rJlog)rPZscheduler_outputsZ model_outputrQrQrR do_log_stats'szAsyncLLM.do_log_statscstd|jr |jdS)NzCalled check_health.)rzdebugrn dead_errorr^rQrQrR check_health/s  zAsyncLLM.check_healthc|jdIdHdS)NTrIZ profile_asyncr^rQrQrR start_profile4zAsyncLLM.start_profilecrrrr^rQrQrR stop_profile7rzAsyncLLM.stop_profilecs2|jj|j|jj|jIdHdSr\)rGr5Zreset_processor_cacher?Zmm_input_cache_clientresetrIZreset_mm_cache_asyncr^rQrQrRreset_mm_cache:s zAsyncLLM.reset_mm_cachedevicecs(|tjkr td|jIdHdS)NzNot supported on CPU.)rZCPUrFrIZreset_prefix_cache_async)rPrrQrQrRreset_prefix_cache?s zAsyncLLM.reset_prefix_cachelevelcs&|IdH|j|IdHdSr\)rrIZ sleep_async)rPrrQrQrRrEszAsyncLLM.sleeptagscs|j|IdHdSr\)rIZ wake_up_async)rPrrQrQrRwake_upIrzAsyncLLM.wake_upcrar\)rIZis_sleeping_asyncr^rQrQrR is_sleepingLrczAsyncLLM.is_sleepingc|j|IdHS)zsJ