o )i @sUddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddl m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddlm'Z'm(Z(m)Z)ddl*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7ddl8m9Z9ddl:m;Z;dd lZ>dd l?m@Z@mAZAmBZBmCZCdd lmDZDdd lEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQdd lRmSZSddlmTZTddlUZUddlVZVddlWZWddlXZYddlZmEZ[ddl\Z\ddl]Z^ddl_Z_ddl`Z`ddlaZ`ddlbZbddlcZcddldZcddlemfZfddlgmhZhddlimjZjddlkmlZlddlmmnZnmoZompZpmqZqddlrmsZsddltmuZumvZvddlwmxZxeFrddl myZyddlzm{Z{m|Z|eve}Z~dZdZdZdZdZdZdZdZd Zd!Zd"Zd#Zeeeeeeeeed$ Zd%Zd&ed'<d(Zd&ed)<d*Zd&ed+<d,Zd&ed-<d.Zd&ed/<d0Zd&ed1<d2Zd&ed3<d4Zd&ed5<d6Z d7Z d8Zd9Ze`je`je`je`je`je`je`je`je`jd: Ze`jeYje`jeYje`jeYje`jeYje`jeYje`jeYjiZejdd=d>Zeod?ZeNd@ZeNdAZeNdBe2dCZeNdDZeNdEZGdFdGdGZeZGdHdIdIejZGdJdKdKejZGdLdMdMZGdNdOdOe+eefZGdPdQdQeKZGdRdSdSeUjeefeIeefZGdTdUdUZe@dddXdYZddZd[Zdd\d]ZGd^d_d_ZddbdcZddfdgZddldmZddpdqZ d d!dwdxZd"d{d|Zd#ddZd$ddZÐdddZddZŐdddZƐd%ddZǐd&ddZȐd'ddZɐd(ddZʐd(ddZːdddZ̐dddZ͐dddZΐdddZϐd)ddZАd*ddZѐd+ddZҐd,ddZӐdddZԐd-ddZՐd.ddZ֐d.ddZאd/ddÄZ d d0ddɄZ   ːd1d2dd؄Z   ʐd3d4ddڄZe@d5dd܄Ze@d5ddބZGdddZddd6ddZddddd7ddZd8ddZd9ddZd9ddZd:ddZd;ddZd<ddZddd=ddZd>ddZd?ddZd@ddZe@dAddZdd d!Ze`jjZeZdBd$d%Zee`j_dCd&d'ZdDd*d+ZdEd,d-ZeNd.eHd/eGfdCZ 0 dFdGd6d7Zd0dd8dHd:d;ZeAd<d= d dId?d@ZddAdBZd5dCdDZd5dEdFZ dJdKdJdKZdLdOdPZdMdSdTZGdUdVdVe!ZGdWdXdXe"e%ZGdYdZdZe#ZdNd]d^Zdd0d_dOdedfZd0dd_dPdjdkZd5dldmZd5dndoZd5dpdqZGdrdsdsZ Gdtdudue6e efeIeZ Gdvdwdwe+e eefZ dQdydzZdRd~dZdSddZdTddZe@ddZGdddZGdddeZGdddeZejddZ    dUdVddZdWddZdXddZe=GdddZe=GdddZejdYddZdZddZddZd[ddZd d\ddZ   d]d^dĐdńZ!ej  d_d`dǐdȄZ"dɐdʄZ#dːd̄Z$ d dadѐd҄Z%dbdِdڄZ&dېd܄Z'dcdߐdZ(Gdddej)Z*ddddZ+ejd deddZ,dfdgddZ-dhddZ.dddZ/dddZ0diddZ1djddZ2dkddZ3e@dlddZ4d5ddZ5d5ddZ6d5ddZ7d5ddZ8 dmdnd dZ9doddZ:d dpddZ;dS(q) annotationsN)ActionArgumentDefaultsHelpFormatterArgumentParserArgumentTypeErrorRawDescriptionHelpFormatter_ArgumentGroup)FIRST_COMPLETEDAbstractEventLoopTask)UserDict defaultdict) AsyncGenerator Awaitable Collection GeneratorHashableIterableIteratorKeysViewMappingSequence)ThreadPoolExecutor)ProcessPoolExecutor) dataclassfield)cache lru_cachepartialwraps)MappingProxyType) TYPE_CHECKINGAnyCallableGenericLiteral NamedTupleOptionalTextIOTypeVarUnioncastoverload)urlparseuuid4)version)Version)Library) BatchEncoding)Never ParamSpecTypeIs assert_never)enable_trace_function_call init_logger)is_in_ray_actor) Namespace) ModelConfig VllmConfigiiizOSliding window attention for encoder/decoder models is not currently supported.zEPrefix caching for encoder/decoder models is not currently supported.zFChunked prefill for encoder/decoder models is not currently supported.ztModels with logits_soft_cap require FlashInfer backend, which is currently not supported for encoder/decoder models.zd?ZdWd@dAZdXdYdBdCZdWdDdEZZS)ZLRUCacheNcapacityrU getsizeofOptional[Callable[[_V], float]]cs8t||tt|_d|_d|_tddd|_dS)Nrr) rrxsetra pinned_items_hits_totalr _last_info)rwrrrr[r\rxs  zLRUCache.__init__T update_infokeyrarboolrqrccs0t|}|r|jd7_|jd7_|Srz)r __getitem__rr)rwrrvaluerr[r\r"s zLRUCache.__getitem__rrcsN||v}|j|dd}t|||jvr|||r%|||dSdSNFr)rr __delitem__r_unpin _on_remove)rwrZ run_on_removerrr[r\r+s   zLRUCache.__delitem__rcCst|j|jS)z:Return the internal cache dictionary in order (read-only).)rZ _Cache__dataorderrr[r[r\r6szLRUCache.cachercCr)z1Return the internal order dictionary (read-only).)r _LRUCache__orderrr[r[r\r= zLRUCache.ordercC|jSrtmaxsizerr[r[r\rBzLRUCache.capacitycCrr~)rcurrsizerr[r[r\usageFrzLRUCache.usageF)deltarrcCs,t|j|jd}|r||j}||_|}|S)z Gets the cumulative number of hits and queries against this cache. If `delta=True`, instead gets these statistics since the last call that also passed `delta=True`. r)rrrr)rwrinfoZ info_deltar[r[r\statMs  z LRUCache.statcCs2z |j|WdStyd|j|<YdSwrt)r move_to_endKeyErrorrwrr[r[r\touch]s  zLRUCache.touch Optional[_V]cCdSrtr[rr[r[r\getcz LRUCache.getdefault Union[_V, _T]cCrrtr[rwrrr[r[r\rgrOptional[Union[_V, _T]]cCs<||vr|j|dd}|jd7_n|}|jd7_|S)NFrr{)rrrrwrrrr[r[r\rkscCrrtr[rr[r[r\pop|rz LRUCache.popcCrrtr[rr[r[r\rrcCs(||vr|S|j|dd}|||Sr)rrrr[r[r\rs rcCs|||dSrt) __setitem__rwrrr[r[r\putsz LRUCache.putcCs(||vr td|d|j|dS)zd Pins a key in the cache preventing it from being evicted in the LRU order. zCannot pin key: z not in cache.N) ValueErrorraddrr[r[r\pinsz LRUCache.pincCs|j|dS)z_ Unpins a key in the cache allowing it to be evicted in the LRU order. N)rremoverr[r[r\rszLRUCache._unpincCrrtr[rr[r[r\rszLRUCache._on_remove remove_pinnedrcCs t|dkrdS|j|ddS)Nrr)lenpopitem)rwrr[r[r\ remove_oldests zLRUCache.remove_oldestcCs(|j|jkr||j|jksdSdSrt)rrrrr[r[r\_remove_old_if_neededs zLRUCache._remove_old_if_neededcsV|stfddjDt}|turtdnttj}tt|}||fS)z>Remove and return the `(key, value)` pair least recently used.c3s|] }|jvr|VqdSrt)r).0rrr[r\ sz#LRUCache.popitem..z:All items are pinned, cannot remove oldest from the cache.)nextrALL_PINNED_SENTINEL RuntimeErrorrrr+ra)rwrZlru_keyrr[rr\rszLRUCache.popitemcCsBt|dkr|jddt|dksd|_d|_tddd|_dS)NrTrr)rrrrrrrr[r[r\clears   zLRUCache.clearrt)rrUrr)rrarrrqrc)rrarqrr)rqr)rqrr)rrrqr)rrarqr)rrrrarqr)rrrrarqr)rrarqrc)rrarrrqr)rrarrrqr)rrarrcrqrr)rrarrrqrr)rrrqrrrF)rr)rfrgrhrxrrrrrrrrrr,rrrrrrrrrrrr[r[rr\rsF                rc@s0eZdZdZddZddZddZdd Zd S) PyObjectCachez^Used to cache python objects to avoid object allocations across scheduler iterations. cCs4||_d|_g|_tdD] }|j|q dS)Nr) _obj_builder_index _obj_cacherangeappend)rwZ obj_builder_r[r[r\rxs  zPyObjectCache.__init__cCs,t|j}t|D] }|j|q dSrt)rrrrr)rwZnum_objsrr[r[r\ _grow_caches  zPyObjectCache._grow_cachecCsJ|jt|jkr||jt|jksJ|j|j}|jd7_|S)zxReturns a pre-allocated cached object. If there is not enough objects, then the cache size will double. r{)rrrr)rwobjr[r[r\ get_objects  zPyObjectCache.get_objectcCs d|_dS)zMMakes all cached-objects available for the next scheduler iteration. rN)rrr[r[r\rrzPyObjectCache.resetN)rfrgrh__doc__rxrrrr[r[r[r\rs  rgpurqcCs.ddlm}||}|dksJdt|S)zez?AsyncMicrobatchTokenizer._batch_encode_loop..csfddt||DS)Ncs"g|] \}}j|fi|qSr[)r)rpkwrr[r\ hszQAsyncMicrobatchTokenizer._batch_encode_loop....)zip)promptsr rr[r\hs z=AsyncMicrobatchTokenizer._batch_encode_loop..)rrtimerrrrwait_forr TimeoutErrorrrrun_in_executorr enumeratedoneitems set_resultr3r& Exception set_exception)rwrrr r rr'Z kwargs_listresult_futuresdeadlinetimeoutZ encode_fnresultsfutrreser[)r|rwr\rDsj        z+AsyncMicrobatchTokenizer._batch_encode_loopc s8 |IdH\}}|g}|g}|j|j}t||jkrW||j}|dkr,n+zt||IdH\}}||||Wn tj yOYnwt||jks z#|j |j |j j |IdH}t||D] \} } | sx| | qkWn ty} z|D] } | s| | qWYd} ~ nd} ~ wwq)z.Batch incoming decode requests for efficiency.TNr)rrr)rrrrr*rr+r,rrZ batch_decoder&r.r0r1r2) rwrrrZtoken_ids_listr3r4r5r6r7r8r9r[r[r\rwsN       z+AsyncMicrobatchTokenizer._batch_decode_loopoprGr dictcCst|dkrdS|dd}|dd}|d}|sd|dd fSt|jd d }|d us2|d ur8||kr8d|dd fSd S) a Return a normalized key describing operation + kwargs. - `add_special_tokens`: {True/False} - `truncation`: {True/False} - If `truncation` is False (`max_length` is None), returns a key for a can_batch queue. - If `truncation` is True and `max_length` is None or equals `tokenizer.model_max_length`, returns a key for a can_batch queue. - Otherwise, returns a key for a cannot_batch queue. Examples: - Decode: ("decode",) - Encode typical: ("encode", add_special_tokens, bool_truncation, max_length_label) - Fallback: ("encode", "other") r)radd_special_tokensT truncationF max_lengthrNZmodel_max_length model_max)rr)rgetattrr)rwr:r r<r=r>r?r[r[r\r s     z#AsyncMicrobatchTokenizer._queue_keycsNt|ddr!t|dd}r#|s%fdd}||dSdSdSdS)NrrcsD]}|qdSrt)canceltasktasksr[r\ cancel_taskss z6AsyncMicrobatchTokenizer.__del__..cancel_tasks)r@ is_closedcall_soon_threadsafe)rwrrFr[rDr\__del__s  z AsyncMicrobatchTokenizer.__del__N)rr)rrYrrUrqrr)rrrrrqr)rrrr)rr)r:rGr r;rqr) rfrgrhrrxrrr rrr rIr[r[r[r\rs     3 ! $rrCr cCs(|r|st||jdSdSdSrt)r. run_in_loopget_looprArBr[r[r\cancel_task_threadsafes rLsockets/Sequence[Union[zmq.Socket, zmq.asyncio.Socket]]cCs"|D] }|dur|jddqdS)Nrlinger)close)rMsockr[r[r\ close_socketss  rSrr functionr#cGs6t|r ||dS|s|j|g|RdSdSrt)in_looprGrH)rrTargsr[r[r\rJs  rJ event_looprcCs$zt|kWStyYdSw)NF)rrr)rWr[r[r\rUs  rUfuncCallable[P, T]executor%Optional[concurrent.futures.Executor]Callable[P, Awaitable[T]]csd fdd }|S) zTake a blocking function, and run it on in an executor thread. This function prevents the blocking function from blocking the asyncio event loop. The code in this function needs to be thread safe. rVP.argsr P.kwargsrqasyncio.Futurecs,t}tg|Ri|}|j|dS)NrZrX)rget_event_looprr,)rVr rZp_funcr`r[r\_async_wrappersz"make_async.._async_wrapperN)rVr]r r^rqr_r[)rXrZrbr[r`r\ make_asyncs rciteratorAsyncGenerator[T, None]cCs||Srt)r __anext__)rdrr[r[r\ _next_tasksrg iterators#AsyncGenerator[tuple[int, T], None]c st|dkr|d2z 3dHW}d|fVq 6dStfddt|D}zj|rftj|tdIdH\}}|D]'}||}z|IdH}|\}}||t|<||fVWq<t ycYq.) return_when)rrrr-waitrr rrgStopAsyncIterationr/ contextlibsuppress BaseExceptionrAaclose) rhitemZawaitsr.rdrkr|itfr[rlr\merge_async_iteratorssL        rxlist[T]cs(g}|2z 3dHW}||q6|S)z6Collect all items from an async generator into a list.N)r)rdr/rtr[r[r\collect_from_async_generator!s  rzcCstj}dtjvrdtjvrtd|r|Sttjtj}z | d| dWSt y3Ynwzttj tj}| d| dWSt yQYnwt jddd d S) NZHOST_IP VLLM_HOST_IPaThe environment variable HOST_IP is deprecated and ignored, as it is often used by Docker and other software to interact with the container's network stack. Please use VLLM_HOST_IP instead to set the IP address for vLLM processes to communicate with each other.)z8.8.8.8Pr)z2001:4860:4860::8888r|zFailed to get the IP address, using 0.0.0.0 by default.The value can be set by the environment variable VLLM_HOST_IP or HOST_IP. stacklevelz0.0.0.0)envsr{osenvironloggerwarningsocketAF_INET SOCK_DGRAMconnect getsocknamer1AF_INET6warningswarn)host_ipsr[r[r\get_ip*s4    rcCs@zt|tj}||df|WdStyYdSw)NrTF)rrbindrQOSError)addressfamilyrr[r[r\test_loopback_bindRs rcCs6tj}|r|StdtjrdStdtjrdStd)Nz 127.0.0.1z::1zsNeither 127.0.0.1 nor ::1 are bound to a local interface. Set the VLLM_LOOPBACK_IP environment variable explicitly.)rZVLLM_LOOPBACK_IPrrrrr)Z loopback_ipr[r[r\get_loopback_ip\s  rrcCs&zt|WdStyYdSw)NTF) ipaddress IPv6Addressr)rr[r[r\is_valid_ipv6_addressms   r host_porttuple[str, int]cCsZ|dr |dd\}}|dd}|dd}|t|fS|d\}}|t|fS)N[]r{:) startswithrsplitsplitrY)rhostportr[r[r\split_host_portus    rrrcCs&t|r d|d|S|d|S)Nr]:rr)rrr[r[r\join_host_portsripcC t||Srt) get_tcp_urirrr[r[r\get_distributed_init_methodryrcCs(t|r d|d|Sd|d|S)Nztcp://[rztcp://rrrr[r[r\rsrcCstj}d|dtS)Nzipc:///)rZVLLM_RPC_BASE_PATHr/)Z base_rpc_pathr[r[r\get_open_zmq_ipc_pathsrcCs dtS)Nz inproc://r.r[r[r[r\get_open_zmq_inproc_path rcCs:dtjvrtj}t||d} t}||vr|SqtS)a< Get an open port for the vLLM process to listen on. An edge case to handle, is when we run data parallel, we need to avoid ports that are potentially used by the data parallel master process. Right now we reserve 10 ports for the data parallel master process. Currently it uses 2 ports. VLLM_DP_MASTER_PORT )rrrrr_get_open_port)Zdp_master_portZreserved_port_rangeZcandidate_portr[r[r\ get_open_ports rc Cs6tj}|durD z$ttjtj}|d|f|WdWS1s&wYWntyB|d7}td|d|Ynwqz'ttjtj}|d| dWdWS1sdwYWdStyttj tj}|d| dWdYS1swYYdSw)NTr{z)Port %d is already in use, trying port %d)rr) rZ VLLM_PORTrr SOCK_STREAMrrrrrr)rrr[r[r\rs4&    (   *rOptional[psutil.Process]c CsZtjdrdStD]}|jj|kr*z t|jWStj y)YdSwq dS)Ndarwin) sysplatformrrZnet_connectionsladdrrProcesspid NoSuchProcess)rconnr[r[r\find_process_using_ports   rrdict[str, str]cCsN|D] \}}|tjvrtj||krtd|tj|||tj|<qdS)Nz5Overwriting environment variable %s from '%s' to '%s')r/rrrr)rrr r[r[r\update_environment_variabless  rlst chunk_sizeccs.tdt||D] }||||Vq dS)z,Yield successive chunk_size chunks from lst.rN)rr)rrr|r[r[r\ chunk_listsrabcCs ||  S)zCeiling division.r[)rrr[r[r\cdivs rcCs|dkrdSd|d>S)zThe next power of 2 (inclusive)r{ bit_lengthnr[r[r\next_power_of_2rrcCs|dkrdSd|d>S)z#The previous power of 2 (inclusive)rr{rrr[r[r\prev_power_of_2rrxycCs||d||Srzr[rrr[r[r\round_upsrcCs |||Srtr[rr[r[r\ round_downrrtensor torch.TensorlowrUhighrrcCs:ddlm}tj|tjd}||||||~dS)Nrrdtype)rrrZZ empty_likefloat16uniform_Z convert_fp8)rrrrZ tensor_tmpr[r[r\_generate_random_fp8s   r cache_dtype!Optional[Union[str, torch.dtype]] model_dtype torch.dtypecCst|tr:|dkr)t|tr|tvrt|}|St|tjr"|}|Std||tvr3t|}|Std|t|tjrD|}|Std|)NrkzInvalid model dtype: zInvalid kv cache dtype: ) isinstancerGSTR_DTYPE_TO_TORCH_DTYPErZrr)rr torch_dtyper[r[r\get_kv_cache_torch_dtypes(    rcudaNHD num_blocks block_size num_layers num_heads head_sizeseed Optional[int]device Optional[str] cache_layout-tuple[list[torch.Tensor], list[torch.Tensor]]c sddlm} | |t||} |d|||f| dvsJ| dkr#dnd} tfdd | D} |d }g}g}t|D]B}tj| | |d j| }|d vrU| | |n|d krat || |nt d|| |dddf| |dddfq<||fS)Nrcurrent_platformr})rZHNDr)rr{r})rr{rr}rc3s|]}|VqdSrtr[)rr|Zgeneric_kv_cache_shaper[r\rBsz5create_kv_caches_with_random_flash..࿩sizerrrkrSrTrUrV#Does not support key cache of type r{) vllm.platformsrseed_everythingrrrrZemptyZpermuterrrr)rrrrrrrrrrrrZ stride_orderZkv_cache_allocation_shapescale key_caches value_cachesrZkey_value_cacher[rr\"create_kv_caches_with_random_flash-s:     rc CsV|dkr|drtd|ddlm} | |t||} |d} dtjg| d} |||| || f} g}t|D].}tj | | |d}|d vrS| | | n|dkr_t || | ntd || |q=||||f}g}t|D].}tj || |d}|d vr| | | n|dkrt || | ntd || |qx||fS) NrVz6Does not support key cache of type fp8 with head_size rrrrrrrz%Does not support value cache of type ) rrrrrrZr element_sizerrrrr)rrrrrrrrrrrrrZkey_cache_shaperrZ key_cacheZvalue_cache_shaperZ value_cacher[r[r\create_kv_caches_with_randomYsP         rcCsddlm}|SNrr)rris_pin_memory_availablerr[r[r\rs rcCstS)z7Check if Unified Virtual Addressing (UVA) is available.)rr[r[r[r\is_uva_availablesrc@s2eZdZddddZddd Zd d Zd d ZdS)DeviceMemoryProfilerNrOptional[torch.types.Device]cCrsrt)r)rwrr[r[r\rxryzDeviceMemoryProfiler.__init__rqrUcCs ddlm}t||jSr)rrgccollectZget_current_memory_usager)rwrr[r[r\current_memory_usages  z)DeviceMemoryProfiler.current_memory_usagecCs||_|Srt)r initial_memoryrr[r[r\ __enter__s zDeviceMemoryProfiler.__enter__cCs$||_|j|j|_tdSrt)r Z final_memoryrZconsumed_memoryr r )rwexc_typeexc_valexc_tbr[r[r\__exit__s  zDeviceMemoryProfiler.__exit__rt)rr r)rfrgrhrxr rrr[r[r[r\r s    r max_len list[list[T]]padr npt.DTypeLiker npt.NDArraycCsl|dur ttt|dd}tjt||f||d}t|D]\}}t||ks)J|||dt|f<q|S)z Make a padded array from 2D inputs. The padding is applied to the end of each inner list until it reaches `max_len`. Nr)rr)maxmaprnpfullr-)rrrrpadded_xindZblocktbr[r[r\make_ndarray_with_pads r F)rr pin_memory"Optional[Union[str, torch.device]]r!c Cs8t|}t||||d}t||}|r|}|S)z Make a padded tensor from 2D inputs. The padding is applied to the end of each inner list until it reaches `max_len`. r)TORCH_DTYPE_TO_NUMPY_DTYPEr rZZ from_numpytor!) rrrrrr!Znp_dtyperrr[r[r\make_tensor_with_pads r%rlist target_deviceUnion[str, torch.device]cCs tj|||dd}|j|ddS)z?Asynchronously create a tensor and copy it from host to device.cpu)rr!rT)rZ non_blocking)rZrr$)rrr'r!tr[r[r\async_tensor_h2dsr+cCstjg|dS)z'Get the size of the data type in bytes.r)rZrrrr[r[r\get_dtype_sizer,cCs|tjk|j|jdS)Nr})rZris_floating_point is_complexrr[r[r\_get_precision_levelsr0 src_dtype tgt_dtypecCs||krdSt|}t|}||krdS||krdS|js6|js6t|}t|}|j|jko5|j|jkSt|}t|}|j|jkoQ|j|jkoQ|j|jkS)z[ Test whether it is lossless to cast a tensor from `src_dtype` to `tgt_dtype`. TF) r0r.r/rZZiinfominrZfinfo resolution)r1r2Z src_levelZ tgt_levelZsrc_infoZtgt_infor[r[r\is_lossless_casts"      r5dtypesCollection[torch.dtype]cstfdddS)zv Get the common `dtype` where all of the other `dtypes` can be cast to it without losing any information. cstfddDS)Nc3|]}t|VqdSrt)r5)rdtrr[r\r"z?common_broadcastable_dtype....)sumrr6rr\r("sz,common_broadcastable_dtype..r)rr<r[r<r\common_broadcastable_dtypes r> maybe_list Iterable[T]cCst|tr|St|S)z5Convert iterable to list, unless it's already a list.)rr&)r?r[r[r\as_list&srAfirst)checkrobjecttyp#Union[type[T], tuple[type[T], ...]]rCLiteral['first', 'all']TypeIs[list[T]]csZt|tsdS|dkrt|dkpt|dS|dkr'tfdd|DSt|dS)NFrBrallc3r8rt)r)rr rEr[r\r8r:zis_list_of..)rr&rrIr7)rrErCr[rJr\ is_list_of,s  rKlistsIterable[Iterable[T]]cCsdd|DS)z)Flatten a list of lists to a single list.cSsg|] }|D]}|qqSr[r[)rZsublistrtr[r[r\r%?r"z$flatten_2d_lists..r[)rLr[r[r\flatten_2d_lists=srNvalues Iterable[_V]rCallable[[_V], _K]cCs8ttttft}|D] }||||q |S)z[ Unlike [`itertools.groupby`][], groups are not broken by non-contiguous data. )r rar&rcrr/)rOrgroupsrr[r[r\ full_groupbyBsrScCsddlm}|dS)z: Lazy initialization of the Hugging Face modules. rinit_hf_modulesN)Z!transformers.dynamic_module_utilsrUrTr[r[r\init_cached_hf_modulesQs  rVlib_namecsjtddg}fdd|D}tj}|s'|r'fdd|dD}|s1tdd|d S) z Find the library file in the system. `lib_name` is full filename, with both prefix and suffix. This function resolves `lib_name` to the full path of the library. z/sbin/ldconfigz-pcs g|] }|vr|dqS))rrlinerWr[r\r%gs z find_library..cs0g|]}tjtj|rtj|qSr[)rpathexistsjoin)rdirr[r[r\r%ks  rz Cannot find z in the system.r) subprocess check_outputr splitlinesrZLD_LIBRARY_PATHrr)rWZlibslocsZenv_ld_library_pathr[r[r\ find_libraryYs  rdcCsVtj}|r td||Stjjdurd}n tjjdurd}ntdtd||S)a We either use the library file specified by the `VLLM_NCCL_SO_PATH` environment variable, or we find the library file brought by PyTorch. After importing `torch`, `libnccl.so.2` or `librccl.so.1` can be found by `ctypes` automatically. z9Found nccl from environment variable VLLM_NCCL_SO_PATH=%sNz libnccl.so.2z librccl.so.1z*NCCL only supports CUDA and ROCm backends.zFound nccl from library %s) rZVLLM_NCCL_SO_PATHrrrZr0rZhipr)Zso_filer[r[r\find_nccl_libraryus   restreamtorch.cuda.StreamcCs|t_t|dSrt)_current_stream_tlsrprev_set_stream)rfr[r[r\_patched_set_streams rjcCsDddlm}ttdrtjdur|rtjntj t_tjS)a replace `torch.cuda.current_stream()` with `vllm.utils.current_stream()`. it turns out that `torch.cuda.current_stream()` is quite expensive, as it will construct a new stream object at each call. here we patch `torch.cuda.set_stream` to keep track of the current stream directly, so that we can avoid calling `torch.cuda.current_stream()`. the underlying hypothesis is that we do not call `torch._C._cuda_setStream` from C/C++ code. rrrN) rrhasattrrhris_rocmrZrZStreamcurrent_streamrr[r[r\rms   rm vllm_configr=cCstjrEt}tj|t}dt dt dt j ddd}tj|dd|j|}tjtj|d d t|d Sd S) ztSet up function tracing for the current thread, if enabled via the VLLM_TRACE_FUNCTION environment variable Z VLLM_TRACE_FUNCTION_for_process_Z_thread_Z_at_z.log rrzvllm-instance-T)exist_okN)rZVLLM_TRACE_FUNCTIONtempfile gettempdirrr\r^getpassgetusergetpid threading get_identdatetimenowreplaceZ instance_idmakedirsdirnamer8)rnZtmp_dirfilenamelog_pathr[r[r\%enable_trace_function_call_for_threads"    rcKs|S)z!Returns the first provided value.r[)rr r[r[r\identityrrF.T start_index is_deprecatedUnion[bool, Callable[[], bool]]additional_messageCallable[[F], F]cs(ts ttdfdd }|S)NfnrrqcsRtj}tjjtjjffdd|Dtfdd}|S)Ncsg|] \}}|jvr|qSr[)kind)rr$param) pos_typesr[r\r%sz3deprecate_args..wrapper..cs\r't|}|r'd|d}dur|d7}tjt|dd|i|S)NzThe positional arguments 7 are deprecated and will be removed in a future update.rorr~)rrrDeprecationWarning)rVr Zdeprecated_argsmsg)rrrpos_kwsrr[r\inners z.deprecate_args..wrapper..inner)inspect signature parameters ParameterPOSITIONAL_ONLYPOSITIONAL_OR_KEYWORDr/r)rparamsrrrr)rrrr\wrappers  zdeprecate_args..wrapperrrrqr)callablerr)rrrrr[rr\deprecate_argss r)rrkwscs0t|ts ttdfdd }|S)Nrrrqcstfdd}|S)NcsXr%|@}|r%d|d}dur|d7}tjt|dd|i|S)NzThe keyword arguments rrorr~)rrrr)rVr Zdeprecated_kwargsr)rdeprecated_kwsrrr[r\rs  z0deprecate_kwargs..wrapper..innerr)rrrrr)rr\rsz!deprecate_kwargs..wrapperr)rrrr)rrrrr[rr\deprecate_kwargss  rrcuda_visible_devicescCsvddl}ddl}ddlm}|jsdS|r't|jdr$|jnd}n|j }|dkr7|j }|S|}|S)Nrr_device_count_amdsmirX) Z torch.cudaZ torch.versionrrr _is_compiledrlrkrZ_device_count_nvml_CZ_cuda_getDeviceCount)rrZrZ raw_countrr[r[r\_cuda_device_count_statelesss     rcCs ttjS)zGet number of CUDA devices, caching based on the value of CUDA_VISIBLE_DEVICES at the time of call. This should be used instead of torch.cuda.device_count() unless CUDA_VISIBLE_DEVICES has already been set to the desired value.)rrZCUDA_VISIBLE_DEVICESr[r[r[r\cuda_device_count_stateless7s rcCtjsdStjS)zCheck if CUDA is initialized.F)rZrris_initializedr[r[r[r\cuda_is_initializedD  rcCr)zCheck if XPU is initialized.F)rZZxpurrr[r[r[r\xpu_is_initializedKrrnames Sequence[str]tuple[Any, ...]csz|strtj|tfdd|DStd}td|d}|t ||d WdS1s6wYdS)z_Get specified CUDA device property values without initializing CUDA in the current process.c3s|]}t|VqdSrt)r@)rnamepropsr[r\rYr:z-cuda_get_device_properties..forkr{)rZ mp_contextTN) rrZrZget_device_propertiesrmultiprocessing get_contextrsubmitcuda_get_device_propertiesresult)rrZ init_cudaZmp_ctxrZr[rr\rRs    $r bound_methodCallable[..., Any]Callable[..., None]cs&t|j|jdfdd }|S)zzMake an instance method that weakly references its associated instance and no-ops once that instance is collected.rqrrcs(}r|g|Ri|dSdSrtr[)rVr instrefunboundr[r\ weak_boundis zweak_bind..weak_boundNr)weakrefr__self____func__)rrr[rr\ weak_bindbs rrwCallable[P, None]cs$d fdd d _t_S) NrVr]r r^rqrrcs^jrdSjjsd_|i|WdSWddS1s(wYdSNT)has_runlockrVr rwrr[r\rrs "zrun_once..wrapperF)rVr]r r^rqrr)rrvLockr)rwr[rr\run_onceps rc@seZdZdddZdS) StoreBooleanNcCsL|dkrt||jddS|dkrt||jddStd|d)NtrueTfalseFzInvalid boolean value: z. Expected 'true' or 'false'.)lowersetattrdestr)rwparser namespacerO option_stringr[r[r\rs  zStoreBoolean.__call__rt)rfrgrhrr[r[r[r\rsrcs(eZdZdZddZfddZZS)SortedHelpFormatterzASortedHelpFormatter that sorts arguments by their option strings.csDtd}td}|d|}t||}tfdd|DgS)z 1. Sentences split across lines have their single newlines removed. 2. Paragraphs and explicit newlines are split into separate lines. 3. Each line is wrapped to the specified width (width of terminal). z(?.)recompilesubrr;)rwtextrZsingle_newlineZmultiple_newlineslinesr[rr\ _split_liness    z SortedHelpFormatter._split_linescs t|ddd}t|dS)NcSrrt)option_strings)rr[r[r\r(sz3SortedHelpFormatter.add_arguments..r=)sortedr add_arguments)rwactionsrr[r\rsz!SortedHelpFormatter.add_arguments)rfrgrhrrrrr[r[rr\rs rcseZdZUdZeZded<dZded<fddZe j d kr8d$fd d Z fd dZ Gddde ZddZd%fdd Z d$d&fdd ZddZd'dd Zd(d"d#ZZS))FlexibleArgumentParserz=ArgumentParser that allows both underscore and dash in names.z set[Action] _deprecateda}When passing JSON CLI arguments, the following sets of arguments are equivalent: --json-arg '{"key1": "value1", "key2": {"key3": "value2"}}' --json-arg.key1 value1 --json-arg.key2.key3 value2 Additionally, list elements can be passed individually using +: --json-arg '{"key4": ["value3", "value4", "value5"]}' --json-arg.key4+ value3 --json-arg.key4+='value4,value5' rG _json_tipcs4d|vrt|d<|dd|_tj|i|dS)Nformatter_class add_json_tipT)rrrrrxrwrVr rr[r\rxszFlexibleArgumentParser.__init__)r Ncsl|dur d|vr tdt||\}}tjD]}t||j}r1t|||j kr1td|q||fS)Nz--disable-log-requestsz{argument '--disable-log-requests' is deprecated and replaced with '--enable-log-requests'. This will be removed in v0.12.0.zargument '%s' is deprecated) rZ warning_oncerparse_known_argsrrrkrr@r)rwrVractionrrr[r\rs  z'FlexibleArgumentParser.parse_known_argsc2|dd}tj|i|}|rtj||SN deprecatedFrr add_argumentrrrrwrVr rrrr[r\r  z#FlexibleArgumentParser.add_argumentcseZdZfddZZS)z-FlexibleArgumentParser._FlexibleArgumentGroupcrrrrrr[r\rrz:FlexibleArgumentParser._FlexibleArgumentGroup.add_argument)rfrgrhrrr[r[rr\_FlexibleArgumentGroupsrcOs(|j|g|Ri|}|j||Srt)r_action_groupsr)rwrVr groupr[r[r\add_argument_groups z)FlexibleArgumentParser.add_argument_grouprqcs2|jpd}|jr|tjstj||_tSNr)epilogrrrrr format_help)rwrrr[r\rs    z"FlexibleArgumentParser.format_helprVlist[str] | NonerNamespace | Nonec s<|dur tjdd}|r"|ddkr"tdd|D}|r"tdd|vr+||}d.d d }td}tt}t |D]\}}| drtd|vrf| dd\} |j |dd| d| q>|j ||dd| q>| dr|dkr|ddkr|ddkr|ddn|dd} | d| q>|dkr|dt|kr||ddvr| dq>| |q>d/dd} d0fd"d# tttttttfft} tt} t |D]\}}|vrq| d$rhd|vrhd|vr | dd\}}d|vr qn ||d}|d|d%r/|dd&}tt| d'}| d^}zt|} WntjjyK|} Ynw| || }| |}| fd(d)|DO} |qfd*d+t |D}| rtd,d-| | D]\}}| || t|qt ||S)1Nr{rservecs|]}|dkVqdSz--modelNr[rargr[r[r\rz4FlexibleArgumentParser.parse_args..zWith `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option.--configmatchre.MatchrqrGcSs|dddS)z7Replaces underscores with dashes in the matched string.rr-)rrz)rr[r[r\replr-z/FlexibleArgumentParser.parse_args..replz (?<=--)[^\.]*--=)countz-Or}rrz -O.level=>2013z-O.levelr list[str]rdict[str, Any]cSs|}t|D]}||i}q|S)zCreates a nested dictionary from a list of keys and a value. For example, `keys = ["a", "b", "c"]` and `value = 1` will create: `{"a": {"b": {"c": 1}}}` )reversed)rr nested_dictrr[r[r\create_nested_dicts  z=FlexibleArgumentParser.parse_args..create_nested_dictoriginalupdateset[str]cstt}|D]G\}t|tr-t|tr-||}|fdd|DO}q t|trCt|trC||7<q |vrL|||<q |S)zRecursively updates a dictionary with another dictionary. Returns a set of duplicate keys that were overwritten. ch|] }d|qSrr[rrurr[r\ 1r"zSFlexibleArgumentParser.parse_args..recursive_dict_update..)rrGr/rr;rr&r)rr duplicatesr Znested_duplicates)recursive_dict_updaterr\r&s   z@FlexibleArgumentParser.parse_args..recursive_dict_updater+rX,crrr[rr=r[r\rYr"z4FlexibleArgumentParser.parse_args..csg|] \}}|vr|qSr[r[)rr|r)deleter[r\r%\sz5FlexibleArgumentParser.parse_args..zFound duplicate keys %sz, )rrrqrG)rrrrGrqr)rrrrrqr)!rargvanyr_pull_args_from_configrrr&rGr-rrrrrrrYr r;r"rendswithjsondumpsloadsdecoderJSONDecodeErrorrrr^r/r parse_args)rwrVrZmodel_in_cli_argsr patternprocessed_argsr|rrlevelrZ dict_argsrZ processed_argZ value_strrZarg_dictZarg_duplicatesZdict_argZ dict_valuer)r"rrr\r,s      $(              z!FlexibleArgumentParser.parse_argscCsRzt|}Wntyd}t|dwd|kr"dks'tdtd|S)NzPort must be an integeriz#Port must be between 1024 and 65535)rYrr)rwrrr[r[r\ check_portis   z!FlexibleArgumentParser.check_portrcCs0|ddks Jd|d}|t|dkrtd||d}||}|ddkrt|dko:|dd }tdd |D}|sL|sLtd |rj|dg|dg||d |||d d }|S|dg||d|||d d }|S|dg||d|||d d }|S) aMethod to pull arguments specified in the config file into the command-line args variable. The arguments in config file will be inserted between the argument list. example: ```yaml port: 12323 tensor-parallel-size: 4 ``` ```python $: vllm {serve,chat,complete} "facebook/opt-12B" --config config.yaml -tp 2 $: args = [ "serve,chat,complete", "facebook/opt-12B", '--config', 'config.yaml', '-tp', '2' ] $: args = [ "serve,chat,complete", "facebook/opt-12B", '--port', '12323', '--tensor-parallel-size', '4', '-tp', '2' ] ``` Please note how the config args are inserted after the sub command. this way the order of priorities is maintained when these are args parsed by super(). rr{z$More than one config file specified!z`No config file specified! Please check your command-line arguments.rrrcsrrr[rr[r[r\rrz@FlexibleArgumentParser._pull_args_from_config..z]No model specified! Please specify model either as a positional argument or in a config file.r}N)r indexrr_load_config_filerr$)rwrVr2 file_pathZ config_argsZ model_in_cliZmodel_in_configr[r[r\r%usP"       *z-FlexibleArgumentParser._pull_args_from_configr4c Cs|dd}|dvrtd|g}i}zt| }t|}Wdn1s)wYWntyC}ztd||d}~wwdd|jD}| D]%\}} t | t rg||vrg| rf| d |qP| d || t | qP|S) acLoads a yaml file and returns the key value pairs as a flattened list with argparse like pattern ```yaml port: 12323 tensor-parallel-size: 4 ``` returns: processed_args: list[str] = [ '--port': '12323', '--tensor-parallel-size': '4' ] rrX)yamlZymlzPConfig file must be of a yaml/yml type. %s suppliedNzOUnable to read the config file at %s. Make sure path is correctcSsg|] }t|tr|jqSr[)rrr)rrr[r[r\r%s z.r )rropenr5Z safe_loadr1rerror_actionsr/rrrrG) rwr4 extensionr.config config_fileexZstore_boolean_argumentsrrr[r[r\r3s@   z(FlexibleArgumentParser._load_config_file)NNrqrG)rVrrr)rVrrqr)r4rGrqr)rfrgrhrrrrrrxr version_inforrrrrrr,r1r%r3rr[r[rr\rs&      Krr asyncio.Lockc sR|4IdH||i|IdHWdIdHS1IdHs"wYdS)z,Utility function to run async task in a lockNr[)rCrrVr r[r[r\_run_task_with_locks0r@requires_kw_onlyallow_var_kwargsrCallable[..., object]kw_namerBrCc Cst|j}|s dS||}ttjjtjjtjjf}|r?|j |v}|r0|r0|j tjjkr0dS|r9|j tjjks=|s?|r?dS|rU|t t |}|j tjj koT|j |kSdS)zCheck if a keyword is a valid kwarg for a callable; if requires_kw_only disallows kwargs names that can also be positional arguments. FT)rrrrrrrr KEYWORD_ONLYrrr VAR_KEYWORDr) rrErBrCrZ param_valZpassable_kw_typesZ is_sig_paramZ last_paramr[r[r\ supports_kws4   rH overridesOptional[Mapping[str, object]]rcsZ|siSfdd|D}||}|r+r%td||Std||S)a Given a callable which has one or more keyword only params and a dict mapping param names to values, drop values that can be not be kwarg expanded to overwrite one or more keyword-only args. This is used in a few places to handle custom processor overrides for multimodal models, e.g., for profiling when processor options provided by the user may affect the number of mm tokens per instance. Args: callable: Callable which takes 0 or more keyword only arguments. If None is provided, all overrides names are allowed. overrides: Potential overrides to be used when invoking the callable. allow_var_kwargs: Allows overrides that are expandable for var kwargs. Returns: Dictionary containing the kwargs to be leveraged which may be used to overwrite one or more keyword only arguments when invoking the callable. cs&i|]\}}t|dr||qS)rA)rH)rZ kwarg_namevalrCrrBr[r\r!Csz4get_allowed_kwarg_only_overrides..zRThe following intended overrides are not keyword-only args and will be dropped: %szMThe following intended overrides are not keyword args and will be dropped: %s)r/rrr)rrIrBrCZfiltered_overridesZ dropped_keysr[rLr\ get_allowed_kwarg_only_overrides$s$ rMcCstttjj}|tdkS)Nz2.4.0)r1rZ __version__ base_version)Zbase_torch_versionr[r[r\supports_dynamo^s rPcCstdotjS)Nz 2.8.0.dev)is_torch_equal_or_newerrZ distributedZis_xccl_availabler[r[r[r\ supports_xcclds rScCs ttjdS)NZ custom_op)rkrZlibraryr[r[r[r\supports_custom_opkrrUc@s:eZdZdZd ddZdddZddd Zed d Zd S) AtomicCounterzAn atomic, thread-safe counterrcCs||_t|_dS)z6Initialize a new atomic counter to given initial valueN)_valuervr_lock)rwinitialr[r[r\rxrszAtomicCounter.__init__r{cCs>|j|j|7_|jWdS1swYdS)z@Atomically increment the counter by num and return the new valueNrXrWrwnumr[r[r\incw$zAtomicCounter.inccCs>|j|j|8_|jWdS1swYdS)z@Atomically decrement the counter by num and return the new valueNrZr[r[r[r\dec}r^zAtomicCounter.deccCrrt)rWrr[r[r\rrzAtomicCounter.valueNrrj) rfrgrhrrxr]r_rrr[r[r[r\rVos   rVc@s:eZdZdddZdd d Zdd dZddZddZdS)LazyDictfactorydict[str, Callable[[], T]]cCs||_i|_dSrt)_factory_dict)rwrar[r[r\rxs zLazyDict.__init__rrGrqr_cCs8||jvr||jvrt||j||j|<|j|Srt)rdrcrrr[r[r\rs   zLazyDict.__getitem__rCallable[[], T]cCs||j|<dSrt)rcrr[r[r\rrzLazyDict.__setitem__cCrrt)rrcrr[r[r\rryzLazyDict.__iter__cCrrt)rrcrr[r[r\__len__ryzLazyDict.__len__N)rarb)rrGrqr_)rrGrre)rfrgrhrxrrrrfr[r[r[r\r`s     r`c@s0eZdZdddZdd d Zd d dddZdS) ClassRegistryrtype[T]rqrccCs.|D]}||jvr|j|Sqt|rt)mrorr)rwrclsr[r[r\rs  zClassRegistry.__getitem__rDrcCs ||Srt)containsrr[r[r\ __contains__ryzClassRegistry.__contains__F)strictrmcs6t|tsdS|r|jvStfdd|DS)NFc3s|]}|jvVqdSrt)r)rrjrr[r\rr:z)ClassRegistry.contains..)rtyperr$ri)rwrrmr[rr\rks  zClassRegistry.containsN)rrhrqrc)rrDrqr)rrDrmrrqr)rfrgrhrrlrkr[r[r[r\rgs  rgr"cCst|tjr tjj|S|S)z Create a weak reference to a tensor. The new tensor will share the same data as the original tensor, but will not keep the original tensor alive. )rrZTensorrrweak_ref_tensor)rr[r[r\rps rptensors.css|]}t|VqdSrtrtrur[r[r\rrz#weak_ref_tensors..zInvalid type for tensors)rrZrorpr&rr)rqr[r[r\weak_ref_tensorss   rv cpu_tensorcCs|sJdtjj|S)zQ Get a CUDA view of a CPU tensor using Unified Virtual Addressing (UVA). zCPU tensor must be pinned) is_pinnedrZrrget_cuda_view_from_cpu_tensor)rwr[r[r\rysry module_namer4Union[str, os.PathLike]cCsZtj||}|durtd|d|jdusJtj|}|tj|<|j||S)z Import a Python file according to its file path. Based on the official recipe: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly NzNo module named '') importlibutilspec_from_file_locationModuleNotFoundErrorloadermodule_from_specrmodules exec_module)rzr4specmoduler[r[r\import_from_paths   rcs6tjd}|dg|dg}fdd|DS)Nrz Requires-DistzProvides-Extracs i|] fddDqS)cs.g|]}|ddrtd|dqS)z extra == ""z ;|>=|<=|==r)r&rr)rreqextrar[r\r%s z=get_vllm_optional_dependencies...r[)r requirementsrr\r!s  z2get_vllm_optional_dependencies..)r}metadataget_all)rextrasr[rr\get_vllm_optional_dependenciess    rc@seZdZdZd]ddZd^d d Zd^d d Zd^ddZd^ddZd^ddZ d^ddZ ddZ ddZ d_ddZ ddZd`d d!Zdad#d$Zd`d%d&Zd^d'd(Zd^d)d*Zd^d+d,Zd^d-d.Zd^d/d0Zd^d1d2Zd^d3d4Zd^d5d6Zdbdcd9d:Zd^d;d<Zd^d=d>Zd^d?d@Zd^dAdBZd^dCdDZdEdFZ dGdHZ!dIdJZ"dKdLZ#dMdNZ$dbdddPdQZ%dRdSZ&dTdUZ'dVdWZ(dXdYZ)d_dZd[Z*d\S)e_PlaceholderBaseaP Disallows downstream usage of placeholder modules. We need to explicitly override each dunder method because [`__getattr__`][vllm.utils._PlaceholderBase.__getattr__] is not called when they are accessed. Info: [Special method lookup](https://docs.python.org/3/reference/datamodel.html#special-lookup) rrGrqr4cCst)z The main class should implement this to throw an error for attribute accesses representing downstream usage. )NotImplementedErrorrr[r[r\ __getattr__ sz_PlaceholderBase.__getattr__rrDcC |dS)N__lt__rrr[r[r\r ryz_PlaceholderBase.__lt__cCr)N__le__rrr[r[r\r ryz_PlaceholderBase.__le__cCr)N__eq__rrr[r[r\r ryz_PlaceholderBase.__eq__cCr)N__ne__rrr[r[r\r ryz_PlaceholderBase.__ne__cCr)N__gt__rrr[r[r\r ryz_PlaceholderBase.__gt__cCr)N__ge__rrr[r[r\r ryz_PlaceholderBase.__ge__cCr)N__hash__rrr[r[r\r# ryz_PlaceholderBase.__hash__cCr)N__bool__rrr[r[r\r& ryz_PlaceholderBase.__bool__rVr cOr)Nrrrr[r[r\r+ ryz_PlaceholderBase.__call__cCr)Nrfrrr[r[r\rf0 ryz_PlaceholderBase.__len__cCr)Nrrrr[r[r\r3 ryz_PlaceholderBase.__getitem__rcCr)Nrrrr[r[r\r6 ryz_PlaceholderBase.__setitem__cCr)Nrrrr[r[r\r9 ryz_PlaceholderBase.__delitem__cCr)N__add__rrr[r[r\rD ryz_PlaceholderBase.__add__cCr)Nrrrr[r[r\rG ryz_PlaceholderBase.__sub__cCr)N__mul__rrr[r[r\rJ ryz_PlaceholderBase.__mul__cCr)N __matmul__rrr[r[r\rM ryz_PlaceholderBase.__matmul__cCr)N __truediv__rrr[r[r\rP ryz_PlaceholderBase.__truediv__cCr)N __floordiv__rrr[r[r\rS ryz_PlaceholderBase.__floordiv__cCr)N__mod__rrr[r[r\rV ryz_PlaceholderBase.__mod__cCr)N __divmod__rrr[r[r\rY ryz_PlaceholderBase.__divmod__.modulocCr)N__pow__r)rwrrr[r[r\r\ ryz_PlaceholderBase.__pow__cCr)N __lshift__rrr[r[r\r_ ryz_PlaceholderBase.__lshift__cCr)N __rshift__rrr[r[r\rb ryz_PlaceholderBase.__rshift__cCr)N__and__rrr[r[r\re ryz_PlaceholderBase.__and__cCr)N__xor__rrr[r[r\rh ryz_PlaceholderBase.__xor__cCr)N__or__rrr[r[r\rk ryz_PlaceholderBase.__or__cCr)N__neg__rrr[r[r\rq ryz_PlaceholderBase.__neg__cCr)N__pos__rrr[r[r\rt ryz_PlaceholderBase.__pos__cCr)N__abs__rrr[r[r\rw ryz_PlaceholderBase.__abs__cCr)N __invert__rrr[r[r\rz ryz_PlaceholderBase.__invert__cCr)N __index__rrr[r[r\r ryz_PlaceholderBase.__index__ndigitscCr)N __round__r)rwrr[r[r\r ryz_PlaceholderBase.__round__cCr)N __trunc__rrr[r[r\r ryz_PlaceholderBase.__trunc__cCr)N __floor__rrr[r[r\r ryz_PlaceholderBase.__floor__cCr)N__ceil__rrr[r[r\r ryz_PlaceholderBase.__ceil__cCr)Nrrrr[r[r\r ryz_PlaceholderBase.__enter__cOr)Nrrrr[r[r\r ryz_PlaceholderBase.__exit__N)rrGrqr4)rrD)rVrDr rD)rrD)rrDrrD).)rrDrrD)rrD)+rfrgrhrrrrrrrrrrrrfrrrrrrrrrrrrrrrrrrrrrrrrrrrrr[r[r[r\rsR                       rcs6eZdZdZdfdd Zdd d Zdd d ZZS)PlaceholderModulez A placeholder object to use when a module does not exist. This enables more informative errors when trying to access attributes of a module that does not exists. rrGrqrrcst||_dSrt)rrx_PlaceholderModule__name)rwrrr[r\rx s  zPlaceholderModule.__init__ attr_pathcCrrt)_PlaceholderModuleAttrrwrr[r[r\placeholder_attr ryz"PlaceholderModule.placeholder_attrrc Csp|j}z t|Wtdty7}ztD]\}}||vr0d|d|d}t||q|d}~ww)NzPlease install vllm[z] for z supportMPlaceholderModule should not be used when the original module can be imported)rr} import_module ImportErrorrr/AssertionError)rwrrexcrrrr[r[r\r s   zPlaceholderModule.__getattr__)rrGrqrrrrGrrG)rfrgrhrrxrrrr[r[rr\r s  rcs2eZdZdfdd Zdd d Zdd d ZZS)rrrrrGrqrrcst||_||_dSrt)rrx_PlaceholderModuleAttr__module!_PlaceholderModuleAttr__attr_path)rwrrrr[r\rx s  z_PlaceholderModuleAttr.__init__cCst|j|jd|S)Nr)rrrrr[r[r\r sz'_PlaceholderModuleAttr.placeholder_attrrcCs t|j|jd|td)Nrr)r@rrrrr[r[r\r sz"_PlaceholderModuleAttr.__getattr__)rrrrGrqrrrr)rfrgrhrxrrrr[r[rr\r s rrZFRAGMENTCUDAr[op_nameop_func mutates_argsr fake_implOptional[Callable] target_libOptional[Library] dispatch_keytagstuple[torch.Tag, ...]c Cstsddlm}|rJddSddl}t|jdr&|jj||d} n ddl}|j j ||} |p5t } | j || |d| j |||d|durS| ||dSdS) a `torch.library.custom_op` can have significant overhead because it needs to consider complicated dispatching logic. This function directly registers a custom op and dispatches it to the CUDA backend. See https://gist.github.com/youkaichao/ecbea9ec9fc79a45d2adce1784d7a9a5 for more details. By default, the custom op is registered to the vLLM library. If you want to register it to a different library, you can pass the library object to the `target_lib` argument. IMPORTANT: the lifetime of the operator is tied to the lifetime of the library object. If you want to bind the operator to a different library, make sure the library object is alive when the operator is used. rrzcuda platform needs torch>=2.4 to support custom op, chances are you are using an old version of pytorch or a custom build of pytorch. It is recommended to use vLLM in a fresh new environment and let it install the required dependencies.N infer_schema)r)r)r)rUrrZ is_cuda_alike torch.libraryrkrTrZtorch._custom_op.implZ _custom_opimplvllm_libdefineZ_register_fake) rrrrrrrrrZZ schema_strZmy_libr[r[r\direct_register_custom_op s&   rqualnamecCs$|dd\}}t|}t||S)z> Resolve an object by its fully-qualified class name. rr{)rr}rr@)rrzobj_namerr[r[r\resolve_obj_by_qualname s  rrc Cszt|}Wn tjyYdSw|jdd}|D]}ttt|j t j Wdn1s5wYqttt|t j WddS1sSwYdS)z Kills all descendant processes of the given pid by sending SIGKILL. Args: pid (int): Process ID of the parent process NT) recursive) rrrchildrenrprqProcessLookupErrorrkillrsignalSIGKILL)rparentrchildr[r[r\kill_process_tree s   "rc@seZdZUdZdZded<dZded<dZded<dZded<dZ ded<dZ ded <d Z d ed <d Z ded<ddZ ddZdddZdS)MemorySnapshotzMemory snapshot.rrY torch_peak free_memory total_memory cuda_memory torch_memorynon_torch_memoryrU timestampTr auto_measurecCs|jr |dSdSrt)rmeasurerr[r[r\ __post_init__1 s zMemorySnapshot.__post_init__cCs\tjdd|_tj\|_|_|j|j|_tj |_ |j|j |_ t |_ dS)Nzallocated_bytes.all.peakr)rZrZ memory_statsrrZ mem_get_inforrrZmemory_reservedrrr)rrr[r[r\r5 s  zMemorySnapshot.measurerrqc CsPt|j|j|j|j|j|j|j|j|j|j|j|j|j|jddS)NF)rrrrrrrr)rrrrrrrrrr[r[r\rI s       zMemorySnapshot.__sub__N)rrrqr)rfrgrhrrrrrrrrrrrrrr[r[r[r\r% s         rc@seZdZUdZdZded<dZded<dZded<dZded<e e d Z d ed <e e d Z d ed <e e d Z d ed <dZded<dddZdS)MemoryProfilingResultz7Memory profiling result. All numbers are in bytes. rrYnon_kv_cache_memorytorch_peak_increasenon_torch_increaserUweights_memory)default_factoryr before_createbefore_profile after_profiler profile_timerqrGc CsHd|jdd|jtdd|jtdd|jtdd|jtdd S)NzMemory profiling takes z.2fz% seconds. Total non KV cache memory: z!GiB; torch peak memory increase: z(GiB; non-torch forward increase memory: zGiB; weights memory: zGiB.)rr GiB_bytesrrrrr[r[r\__repr__c s     zMemoryProfilingResult.__repr__Nr=)rfrgrhrrrrrrrrrrrrrr[r[r[r\rV s      rbaseline_snapshotr,Generator[MemoryProfilingResult, None, None]ccsttjtjt}||_||_|j |Vttj|j |j |j }|j |j}|j |_ |j|_|j|_|j|j |j|_dS)aMemory profiling context manager. baseline_snapshot: the memory snapshot before the current vLLM instance. weights_memory: memory used by PyTorch when loading the model weights. Note that, before loading the model weights, we also initialize the device and distributed environment, which may consume some memory. This part is not included in the weights_memory because PyTorch does not control it. The memory in one GPU can be classified into 3 categories: 1. memory used by anything other than the current vLLM instance. 2. memory used by torch in the current vLLM instance. 3. memory used in the current vLLM instance, but not by torch. A quantitive example: Before creating the current vLLM instance: category 1: 1 GiB category 2: 0 GiB category 3: 0 GiB After creating the current vLLM instance and loading the model, (i.e. before profiling): category 1: 1 GiB category 2: 2 GiB (model weights take 2 GiB) category 3: 0.5 GiB (memory used by NCCL) During profiling (peak): category 1: 1 GiB category 2: 4 GiB (peak activation tensors take 2 GiB) category 3: 1 GiB (memory used by NCCL + buffers for some attention backends) After profiling: category 1: 1 GiB category 2: 3 GiB (after garbage-collecting activation tensors) category 3: 1 GiB (memory used by NCCL + buffers for some attention backends) In this case, non-kv cache takes 5 GiB in total, including: a. 2 GiB used by the model weights (category 2) b. 2 GiB reserved for the peak activation tensors (category 2) c. 1 GiB used by non-torch components (category 3) The memory used for loading weights (a.) is directly given from the argument `weights_memory`. The increase of `torch.cuda.memory_stats()["allocated_bytes.all.peak"]` during profiling gives (b.). The increase of `non_torch_memory` from creating the current vLLM instance until after profiling to get (c.). N)r r rZrZ empty_cacheZreset_peak_memory_statsrrrrrrrrrrrrr)rrrZ diff_profileZdiff_from_creater[r[r\memory_profilingn s$2       rr0c Cstjdr tddSddl}|j}||\}}||krEz ||||fWdSt yD}zt d||WYd}~dSd}~wwdS)Nwinz-Windows detected, skipping ulimit adjustment.rzFound ulimit of %s and failed to automatically increase with error %s. This can cause fd limit errors like `OSError: [Errno 24] Too many open files`. Consider increasing with ulimit -n) rrrrrresourceZ RLIMIT_NOFILEZ getrlimitZ setrlimitrr)Ztarget_soft_limitrZ resource_typeZ current_softZ current_hardr9r[r[r\ set_ulimit s&   rcCs&t\}}}dt|||}|Sr)rexc_infor^ tracebackformat_exception)etypertbZerr_strr[r[r\get_exception_traceback srr\tuple[str, str, str]cCst|}|jstd||j}|jpd}t|jpd}|dkr.t||fs.td||dkr;|r;td||||fS)z Split a zmq path into its parts.zInvalid zmq path: rtcp)r-schemerhostnamerGrrI)r\parsedr rrr[r[r\split_zmq_path s   r r cCsF|dur |d|St|r|d|d|S|d|d|S)a8Make a ZMQ path from its parts. Args: scheme: The ZMQ transport scheme (e.g. tcp, ipc, inproc). host: The host - can be an IPv4 address, IPv6 address, or hostname. port: Optional port number, only used for TCP sockets. Returns: A properly formatted ZMQ path string. Nz://z://[rrr)r rrr[r[r\ make_zmq_path s  r ctx'Union[zmq.asyncio.Context, zmq.Context] socket_typerOptional[bool]Optional[bytes]rP%Union[zmq.Socket, zmq.asyncio.Socket]cCsPt}||}|jd}|jd} |dkr | dkr td} nd} |dur0|tjtjtj fv}|tj tj tj fvrH| tjd| tj| |tjtj tj fvr`| tjd| tj| |durk| tj||durv| tj||tjkr| tjdt|\} } } | d krt| r| tjd |r|||S|||S) z9Make a ZMQ socket with the proper bind/connect semantics.rQrrgArXNrTrr{)rrrr availablerYzmqZPUSHZSUBZXSUBZPULLZDEALERZROUTER setsockoptZRCVHWMZRCVBUFZSNDHWMZSNDBUFZIDENTITYZLINGERZXPUBZ XPUB_VERBOSEr rZIPV6rr)rr\rrrrPZmemrZ total_memZ available_memZbuf_sizer rrr[r[r\make_zmq_socket s:       rIterator[zmq.Socket]c cstt}z-z t|||||dVWnty tdYn wW|j|ddSW|j|ddS|j|dw)z Context manager for a ZMQ socket)rrzGot Keyboard Interrupt.rON)rContextrKeyboardInterruptrdebugdestroy)r\rrrPrrr[r[r\zmq_socket_ctx; s    rcCstjddkr dSg}tr ddl}|jtjd<|dtr)|dnt r1|d|rCt d d |dtjd<dSdS) zWCheck if we need to force the use of the `spawn` multiprocessing start method. VLLM_WORKER_MULTIPROC_METHODZspawnNrZ RAY_ADDRESSz&In a Ray actor and can only be spawnedzCUDA is initializedzXPU is initializedzWe must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: %sz; ) rrrr:rayZget_runtime_contextZ gcs_addressrrrrrr^)reasonsrr[r[r\_maybe_force_spawnS s$   r!cCsttj}t|S)aJGet a multiprocessing context with a particular method (spawn or fork). By default we follow the value of the VLLM_WORKER_MULTIPROC_METHOD to determine the multiprocessing method (default is fork). However, under certain conditions, we may enforce spawn and override the value of VLLM_WORKER_MULTIPROC_METHOD. )r!rrrr)Z mp_methodr[r[r\get_mp_contextr s r"kv_cachelist[list[torch.Tensor]]shared_kv_cache_layersOptional[dict[str, str]]c s|duri}ddlmddlmfddD}ttfdd|D}|D](}||}|}t|jt|ksCJt |D] \}} | ||j|<qGq+|duru| D]\}} | |kslJd| j|_q\dSdS) Nr) AttentionTypeextract_layer_indexcsBg|]}t|dr|jjjfvr|jdur|qS) attn_typeN)rkr*ZDECODERZENCODER_DECODERZkv_sharing_target_layer_namer layer_name)r'rr[r\r% s  z!bind_kv_cache..c3s|]}|VqdSrtr[r+r(r[r\r s  z bind_kv_cache..z*v0 doesn't support interleaving kv sharing) Zvllm.attentionr'Z vllm.model_executor.models.utilsr)rrr2rr#r-r/) rr#r%Zlayer_need_kv_cacheZlayer_index_sortedr,Z kv_cache_idxZ forward_ctxveZ ve_kv_cacheZtarget_layer_namer[)r'rr)r\ bind_kv_cache~ s>    r.rmethodUnion[str, bytes, Callable]rV tuple[Any]r cCsnt|trtt||}n"t|tr+zt||}Wnty*td|ddwt||}||i|S)a3 Run a method of an object with the given arguments and keyword arguments. If the method is string, it will be converted to a method using getattr. If the method is serialized bytes and will be deserialized using cloudpickle. If the method is a callable, it will be called directly. zMethod z is not implemented.N) rbytesr cloudpickler)rGr@AttributeErrorr)rr/rVr rXr[r[r\ run_method s    r5cCsddlmm}|S)a Historical comments: libnvml.so is the library behind nvidia-smi, and pynvml is a Python wrapper around it. We use it to get GPU status without initializing CUDA context in the current process. Historically, there are two packages that provide pynvml: - `nvidia-ml-py` (https://pypi.org/project/nvidia-ml-py/): The official wrapper. It is a dependency of vLLM, and is installed when users install vLLM. It provides a Python module named `pynvml`. - `pynvml` (https://pypi.org/project/pynvml/): An unofficial wrapper. Prior to version 12.0, it also provides a Python module `pynvml`, and therefore conflicts with the official one. What's worse, the module is a Python package, and has higher priority than the official one which is a standalone Python file. This causes errors when both of them are installed. Starting from version 12.0, it migrates to a new module named `pynvml_utils` to avoid the conflict. It is so confusing that many packages in the community use the unofficial one by mistake, and we have to handle this case. For example, `nvcr.io/nvidia/pytorch:24.12-py3` uses the unofficial one, and it will cause errors, see the issue https://github.com/vllm-project/vllm/issues/12847 for example. After all the troubles, we decide to copy the official `pynvml` module to our codebase, and use it directly. rN)Zvllm.third_party.pynvmlZ third_partypynvml)r6r[r[r\ import_pynvml sr7rjrhcs:|jd ddtd fdd }t|d ||S) a- A replacement for `abc.ABC`. When we use `abc.ABC`, subclasses will fail to instantiate if they do not implement all abstract methods. Here, we only require `raise NotImplementedError` in the base class, and log a warning if the method is not implemented in the subclass. rwrDc Ssg}t|D].}|drqzt||}t|r|j}Wn ty%Yqwt|}d|vr4||q|rKd |}d|d|}t |dSdS)Nrrr!zMethods z not implemented in ) r_rr@rrr4r getsourcerr^rr)rwZunimplemented_methods attr_nameattrZ attr_funcsrcZ method_namesrr[r[r\find_unimplemented_methods s*       zBwarn_for_unimplemented_methods..find_unimplemented_methodsrqrrcs"|g|Ri||dSrtr[rr<Z original_initr[r\ wrapped_init s z4warn_for_unimplemented_methods..wrapped_initrxN)rwrDr)rxrrn __setattr__)rjr>r[r=r\warn_for_unimplemented_methods s  r@cs@eZdZdZdfdd Zdd d ZdddZdddZZS) LazyLoadera LazyLoader module borrowed from Tensorflow https://github.com/tensorflow/tensorflow/blob/main/tensorflow/python/util/lazy_loader.py with a addition of "module caching". Lazily import a module, mainly to avoid pulling in large dependencies. Modules such as `xgrammar` might do additional side effects, so we only want to use this when it is needed, delaying all eager effects local_namerGparent_module_globalsrrcs&||_||_d|_tt|dSrt) _local_name_parent_module_globals_modulerrxrG)rwrBrCrrr[r\rx szLazyLoader.__init__rqtypes.ModuleTypec CsZzt|j}||j|j<|tj|j<Wnty#}z|dd}~ww|j |j|Srt) r}rrfrErDrrr__dict__r)rwrerrr[r[r\_load# s  zLazyLoader._loadrtr"cCs |jdur ||_t|j|Srt)rFrJr@)rwrtr[r[r\r4 s   zLazyLoader.__getattr__rcCs|jdur ||_t|jSrt)rFrJr_rr[r[r\__dir__9 s   zLazyLoader.__dir__)rBrGrCrrrG)rqrG)rtr"rqr")rqr) rfrgrhrrxrJrrKrr[r[rr\rA s  rA dict[_K, _V]key1key2cCsV||}||}|dur|||<n||d|dur#|||<dS||ddS)z5 Helper function to swap values for two keys N)rr)rrMrNv1v2r[r[r\swap_dict_values? s     rQ save_filec csddl}|}|zdVW||r#|dkr#||dS|jdddS||r;|dkr;||w|jddw)zRun a cprofile Args: save_file: path to save the profile result. "1" or None will result in printing to stdout. rNrZcumtime)sort)cProfileZProfileenabledisableZ dump_statsZ print_stats)rRrTZprofr[r[r\cprofile_contextO s   rWenabledcsdfdd }|S)zDecorator to profile a Python method using cProfile. Args: save_file: Path to save the profile result. If "1", None, or "", results will be printed to stdout. enabled: Set to false to turn this into a no-op rXr#cstfdd}|S)NcsLs |i|St|i|WdS1swYdSrt)rWr)rXrXrRr[r\rq s   $z,cprofile..decorator..wrapperr)rXrrXrR)rXr\ decoratoro szcprofile..decoratorN)rXr#r[)rRrXrZr[rYr\cprofilef s  r[ model_configr<cCst|j}t|ddp9dt|jdgvp9t|dddkp9t|do9t|jtr,|jddp9t|jt o9t|jddS)NZalibiFZBloomForCausalLMZ architecturesZposition_encoding_typer attn_config)Zhf_text_configr@Z hf_configrkrr]r;r)r\cfgr[r[r\check_use_alibi s"      r_cCs(tj|tjd}tjt|ddS)uHash any picklable Python object using SHA-256. The input is serialized using pickle before hashing, which allows arbitrary Python objects to be used. Note that this function does not use a hash seed—if you need one, prepend it explicitly to the input. Args: input: Any picklable Python object. Returns: An integer representing the SHA-256 hash of the serialized input. )protocolbig byteorder)pickler(HIGHEST_PROTOCOLrY from_byteshashlibsha256digest)input input_bytesr[r[r\rh s rhcCs.tj|dd}tjt|dd}|d@S)a; Hash objects using CBOR serialization and SHA-256, then truncate to 64bits. This option is useful for non-Python-dependent serialization and hashing. Args: input: Object to be serialized and hashed. Supported types include basic Python types and complex structures like lists, tuples, and dictionaries. Custom classes must implement CBOR serialization methods. Returns: An integer in the range [0, 2^64-1] representing the lower 64 bits of the SHA-256 hash of the CBOR serialized input. T) canonicalrarbl)cbor2r(rYrfrgrhri)rjrkZ full_hashr[r[r\sha256_cbor_64bit s rn hash_fn_namecCs2|dkrtS|dkr tS|dkrtStd|)zGet a hash function by name, or raise an error if the function is not found. Args: hash_fn_name: Name of the hash function. Returns: A hash function. rhrnbuiltinzUnsupported hash function: )rhrnhashr)ror[r[r\get_hash_fn_by_name srrtargetcCs>z tttj|WStyttjdt|kYSw)zCheck if the installed torch version is >= the target version. Args: target: a version string, like "2.6.0". Returns: Whether the condition meets. rZ) _is_torch_equal_or_newerrGrZrNr1r1r}rr0)rsr[r[r\rQ s   rQ torch_versioncCst|}|t|kSrt)r0parse)rursr[r[r\rt s rtcCstj|duS)zReturn True if *module_name* can be found in the current environment. The result is cached so that subsequent queries for the same module incur no additional overhead. N)r}r~ find_spec)rzr[r[r\ _has_module srxcCtdS)z9Whether the optional `pplx_kernels` package is available.Z pplx_kernelsrxr[r[r[r\has_pplx r{cCry)z4Whether the optional `deep_ep` package is available.Zdeep_eprzr[r[r[r\ has_deep_ep r|r}cCry)z6Whether the optional `deep_gemm` package is available.Z deep_gemmrzr[r[r[r\ has_deep_gemm r|r~cCry)z;Whether the optional `triton_kernels` package is available.Ztriton_kernelsrzr[r[r[r\has_triton_kernels r|rrrsuffixrcCsH|r |d|}|rtd|}ntjd|}t|dS)a Set the current process title to a specific name with an optional suffix. Args: name: The title to assign to the current process. suffix: An optional suffix to append to the base name. append: Whether to append to the existing process title. rz::N) setproctitleZ getproctitlerZVLLM_PROCESS_NAME_PREFIX)rrrr[r[r\set_process_title s  rfiler( worker_namecsDtd|d|dtdjd fdd }d _|_d S) z5Prepend each output line with process-specific prefix(z pid=)rorrGcs|sdSjr d}|d|}dkr>|d7}||||t|kr.d_dS|}|d|}dks||dd_dS)Nr rXr{TF)start_new_linefindr)ridxZnext_idxr file_writeprefixr[r\write_with_prefix s   z&_add_prefix..write_with_prefixTN)rrG)CYANRESETwriter)rrrrr[rr\ _add_prefix s  r process_namecCs<|dur tj}t}ttj||ttj||dS)aV Adds a process-specific prefix to each line of output written to stdout and stderr. This function is intended to be called before initializing the api_server, engine_core, or worker classes, so that all subsequent output from the process is prefixed with the process name and PID. This helps distinguish log output from different processes in multi-process environments. Args: process_name: Optional; the name of the process to use in the prefix. If not provided, the current process name from the multiprocessing context is used. N) r"current_processrrrurrstdoutstderr)rrr[r[r\ decorate_logs1 s  r)rXrYr)rrYrqrYrr=)rCr )rMrN)rr rTr#)rWr rqrrt)rXrYrZr[rqr\)rdrerr rqr )rhrerqri)rdrerqry)rrGrqr)rrGrqr)rrGrrYrqrG)rrGrrYrqrG)rrYrqr)rr)rryrrY)rrYrrYrqrY)rrYrqrY)rrYrrYrqrY)rrrrUrrUrqrr)rrrrrqr)NNrr)rrYrrYrrYrrYrrYrrrrrrrrrrrqr)NNr)rrYrrYrrYrrYrrYrrrrrrrrrqr)rqr) rrrr_rrrrrqr)rrrr_rrrrrr"r!rrqr) rr&rrr'r(r!rrqr)rrrqrY)r1rr2r)r6r7)r?r@rqry)rrDrErFrCrGrqrH)rLrMrqry)rOrPrrQr)rWrGrqrG)rfrgrqrr)rqrg)rnr=rqrr)rr_rqr_)TN)rrYrrrrrqr)rrGrrrrrqr)rrrqrYr)rrrqr)rrrqr)rwrrqr)rCr#rr?) rrDrErGrBrrCrrqr) rrDrIrJrBrrCrrqr)rr"rqr")rqrrrqrs)rwrrqr)rzrGr4r{)NNrr[)rrGrr#rrrrrrrrGrr)rrGrqr")rrY)rrrrYrqr)r0)r\rGrqr)r rGrrGrrrqrG)NNN)rrr\rGrr"rrrrrPrrqr)NrN) r\rGrr"rrrPrYrrrqr)rrr#r$r%r&rqrr) rr"r/r0rVr1r rrqr")rjrhrqrh)rrLrMrarNrarqrr)rRrr)rRrrXr)r\r<rqr)rorGrqr#)rsrGrqr)rurGrsrGrqr)rzrGrqr)rF)rrGrrGrrrqrr)rr(rrGrrYrqrr)rrrqrr(< __future__rr concurrentrprxrjr rsrgr}importlib.metadataimportlib.utilrrr'rrrdrrr`rrqrrvr)rtypesrrrargparserrrrrrr r r collectionsr r collections.abcrrrrrrrrrrconcurrent.futuresrconcurrent.futures.processr dataclassesrr functoolsrrrrr typingr!r"r#r$r%r&r'r(r)r*r+r, urllib.parser-r/Z cachetoolsrmr3numpyrZ numpy.typingZnptrregexrrrZZ torch.typesr5rZ zmq.asyncio packagingr0Zpackaging.versionr1rr2Z$transformers.tokenization_utils_baser3Ztyping_extensionsr4r5r6r7Z vllm.envsrZ vllm.loggerr8r9Zvllm.ray.lazy_utilsr:r;Z vllm.configr<r=rfrZDEFAULT_MAX_NUM_BATCHED_TOKENSZ$POOLING_MODEL_MAX_NUM_BATCHED_TOKENSZ'MULTIMODAL_MODEL_MAX_NUM_BATCHED_TOKENSr>r?r@rArBrCrDrErFZSTR_NOT_IMPL_ENC_DEC_ERR_STRSrHrrIrJrKrLrMrNrPZGB_bytesrrrrRrSrTrUZuint8rWZ float8_e4m3fnrrZfloat64Zint32Zint64r#contextmanagerr]r^r_r`rarcrdrerEnumrirlrorrrrrrrrrLrSrJrUrcrgrxrzrrrrrrrrrrrrrrrrrrrrrrrrrrr r r%r+r,r0r5r>rArKrNrSrVrdrerZ set_streamrilocalrhrjrmrrrrrrrrrrrrrrrr@rHrMrPrSrUrVrGr`rnrgrprvryrrrrrrrrrrrrrrr r rrr!r"r.r5r7r@ ModuleTyperArQrWr[r_rhrnrrrQrtrxr{r}r~rrrrr[r[r[r\s 0   8                      $5&  E       ' (                36     , #   P 1:(& "1 0M;.+3