o )iI@stddlZddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z ddlZddlZddlmZmZddlmZmZddlmZe roddlmZmZdd lmZdd lmZdd l m!Z!dd l"m#Z#n dZdZdZdZdZ!dZ#ee$Z%d e&fddZ'Gdddej(Z)Gdddej(Z*Gdddej(Z+Gddde Z,GdddZ-Gddde-Z.dS)N) timedelta)uname) TYPE_CHECKINGAny NamedTupleOptionalUnion) PrefixStore ProcessGroup)ProcessorInputs PromptType) init_logger) ModelConfig VllmConfig) LoRARequest) PoolingParams)SamplingParams)FlexibleArgumentParserreturncCsddtvS)NZ microsoft )joinrlowerrrd/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/platforms/interface.pyin_wsl$src@seZdZeZeZeZeZeZ eZ eZ eZ eZ eZeZeZeZeZeZeZeZeZeZeZeZeZeZeZeZdS)_BackendN)__name__ __module__ __qualname__enumautoZ FLASH_ATTNZFLASH_ATTN_VLLM_V1ZTRITON_ATTN_VLLM_V1ZXFORMERSZ ROCM_FLASHZROCM_AITER_MLAZROCM_AITER_MLA_VLLM_V1Z ROCM_AITER_FA TORCH_SDPAZ FLASHINFERZFLASHINFER_VLLM_V1Z TRITON_MLAZTRITON_MLA_VLLM_V1ZFLASHMLA_VLLM_V1ZFLASHMLAZ CUTLASS_MLAZPALLASZPALLAS_VLLM_V1ZIPEXZDUAL_CHUNK_FLASH_ATTNZDIFFERENTIAL_FLASH_ATTNZ NO_ATTENTIONZFLEX_ATTENTIONZ TREE_ATTNZXFORMERS_VLLM_V1rrrrr)s4 rc@sLeZdZeZeZeZeZeZ eZ eZ eZ dS) PlatformEnumN) rrrrr CUDAROCMTPUXPUCPUNEURONOOT UNSPECIFIEDrrrrr"Es r"c@s4eZdZeZeZeZeZeZ dS) CpuArchEnumN) rrrrr X86ARMPOWERPCOTHERUNKNOWNrrrrr+Ps  r+c@s:eZdZUeed<eed<defddZdefddZdS) DeviceCapabilitymajorminorrcCs|jd|jS)N.)r2r3selfrrras_version_str\szDeviceCapability.as_version_strcCs.d|jkr dksJJ|jd|jS)z Express device capability as an integer ``. It is assumed that the minor version is always a single digit. r )r3r2r5rrrto_int_szDeviceCapability.to_intN)rrrint__annotations__strr7r9rrrrr1Xs r1c@seZdZUeed<eed<eed<dZeed<dZeed<dZeed <d Z eed <dZ eed <gZ e eed <gZ e eed<dZeeed<ede ejfddZdefddZdefddZdefddZdefddZdefddZdefddZdefd d!Zd"edefd#d$Zdefd%d&Zdefd'd(Z e!d)efd*d+Z"e!dd-ede#fd.d/Z$e!d0e#d1ed2ejd3eed4ed5ed6ed7edefd8d9Z%e! :dd)edee&fd;d<Z'e! :dd=e(e)eefefd)edefd>d?Z*e! :dd=e(e)eefefd)edefd@dAZ+e!dd)edefdBdCZ,e!dd)edefdDdEZ-e!dd)edefdFdGZ.e!dHeedefdIdJZ/e!dKdLZ0e!ddMeeddfdNdOZ1e!dPej2ddfdQdRZ3e! ddSee4ddfdTdUZ5e!dVe6ddfdWdXZ7e!dYeddfdZd[Z8e!d\eddfd]d^Z9e!de:fd_d`Z;e!defdadbZde?fdcddZ@e!defdedfZAe!d2ejde)e?e?ffdgdhZBe!defdidjZCe!defdkdlZDe!defdmdnZEe!defdodpZFe!defdqdrZGe!defdsdtZHe!dejfdudvZIe!defdwdxZJe!dyeKdefdzd{ZLe!dyeKdefd|d}ZMe!defd~dZNe!deOde(ePeQfdeRddfddZSdefddZTdefddZUe!dd)edefddZVe!defddZWe!dedeXdededeYdeZf ddZ[e!d3edefddZ\dS)Platform_enum device_name device_typer' dispatch_keyray_device_keyZ'VLLM_DEVICE_CONTROL_ENV_VAR_PLACEHOLDERdevice_control_env_varZinductorsimple_compile_backend dist_backendsupported_quantizationadditional_env_varsN_global_graph_poolrcCstjtjtjgS)z6Returns the supported dtypes for the current platform.)torchZbfloat16Zfloat16Zfloat32r5rrrsupported_dtypesszPlatform.supported_dtypescC |jtjkSNr>r"r#r5rrris_cuda zPlatform.is_cudacCrLrM)r>r"r$r5rrris_rocmrPzPlatform.is_rocmcCrLrM)r>r"r%r5rrris_tpurPzPlatform.is_tpucCrLrM)r>r"r&r5rrris_xpurPzPlatform.is_xpucCrLrM)r>r"r'r5rrris_cpurPzPlatform.is_cpucCrLrM)r>r"r(r5rrr is_neuronrPzPlatform.is_neuroncCrLrM)r>r"r)r5rrris_out_of_treerPzPlatform.is_out_of_tree prompt_lencCtjSrM)sysmaxsize)r6rWrrrget_max_output_tokensszPlatform.get_max_output_tokenscCs|jtjtjfvS)z1Stateless version of [torch.cuda.is_available][].)r>r"r#r$r5rrr is_cuda_alikeszPlatform.is_cuda_alikecCrLrMrNr5rrris_sleep_mode_availablerPz Platform.is_sleep_mode_available device_idcCsB|jtjvrtj|jdkrtj|jd}||}t|S|S)NrB,)rDosenvironsplitr:)clsr^Z device_idsZphysical_device_idrrrdevice_id_to_physical_device_idsz(Platform.device_id_to_physical_device_idF support_facCrXrM)rr!)rcrerrrget_vit_attn_backendszPlatform.get_vit_attn_backendselected_backend head_sizedtypekv_cache_dtype block_sizeuse_v1use_mlahas_sinkc CdS)z,Get the attention backend class of a device.rBr) rcrgrhrirjrkrlrmrnrrrget_attn_backend_clszPlatform.get_attn_backend_clsrcCro)z:Stateless version of [torch.cuda.get_device_capability][].Nrrcr^rrrget_device_capabilityrqzPlatform.get_device_capability capabilitycCs6|j|d}|dur dSt|tr||kS||kS)a. Test whether this platform is compatible with a device capability. The `capability` argument can either be: - A tuple `(major, minor)`. - An integer ``. (See [`DeviceCapability.to_int`][vllm.platforms.interface.DeviceCapability.to_int]) r^NFrs isinstancetupler9rcrtr^Zcurrent_capabilityrrrhas_device_capability   zPlatform.has_device_capabilitycCs6|j|d}|dur dSt|tr||kS||kS)a3 Test whether this platform has exactly the specified device capability. The `capability` argument can either be: - A tuple `(major, minor)`. - An integer ``. (See [`DeviceCapability.to_int`][vllm.platforms.interface.DeviceCapability.to_int]) ruNFrvryrrris_device_capabilityr{zPlatform.is_device_capabilitycCt)zGet the name of a device.NotImplementedErrorrrrrrget_device_namezPlatform.get_device_namecCr})z.Get the uuid of a device, e.g. the PCI bus ID.r~rrrrrget_device_uuid rzPlatform.get_device_uuidcCr})z*Get the total memory of a device in bytes.r~rrrrrget_device_total_memoryrz Platform.get_device_total_memory enforce_eagercCr})zF Check if the current platform supports async output. r~)rcrrrris_async_output_supportedz"Platform.is_async_output_supportedcCs tjddS)aA device-specific wrapper of `torch.inference_mode`. This wrapper is recommended because some hardware backends such as TPU do not support `torch.inference_mode`. In such a case, they will fall back to `torch.no_grad` by overriding this method. T)mode)rJinference_modercrrrrs zPlatform.inference_modeseedcCs0|durt|tj|t|dSdS)z Set the seed of each random module. `torch.manual_seed` will set seed on all devices. Loosely based on: https://github.com/Lightning-AI/pytorch-lightning/blob/2.4.0/src/lightning/fabric/utilities/seed.py#L20 N)randomrnprJZ manual_seed)rcrrrrseed_everything&s   zPlatform.seed_everythingdevicecCr})z: Set the device for the current platform. r~rcrrrr set_device3rzPlatform.set_deviceparsercCro)a Do some pre-registration or update action for the current platform. This function is called before global VllmConfig is initialized or cli arguments are parsed. It's used for out-of-tree platforms to register or update the configuration. For example, the out-of-tree quantization config can be imported and registered here dynamically. Nr)rcrrrrpre_register_and_update:sz Platform.pre_register_and_update vllm_configcCro)ac Check and update the configuration for the current platform. It can raise an exception if the configuration is not compatible with the current platform, or it can update the configuration to make it compatible with the current platform. The config is passed by reference, so it can be modified in place. Nr)rcrrrrcheck_and_update_configJs z Platform.check_and_update_config model_archcCro)a  Verify whether the current platform supports the specified model architecture. - This will raise an Error or Warning based on the model support on the current platform. - By default all models are considered supported. Nr)rcrrrrverify_model_archWs zPlatform.verify_model_archquantcCs.|jr||jvrt|d|jddSdS)zW Verify whether the quantization is supported by the current platform. z, quantization is currently not supported in r4N)rG ValueErrorr?)rcrrrrverify_quantizationcs zPlatform.verify_quantizationcCsTt}|dvr tjS|ds|drtjS|dr"tjS|r'tjStj S)z Determine the CPU architecture of the current system. Returns CpuArchEnum indicating the architecture type. )x86_64amd64i386i686armZaarchppc) platformmachinerr+r, startswithr-r.r/r0)rcrrrrget_cpu_architecturens  zPlatform.get_cpu_architecturecCstr tddSdS)z?Checks whether pin memory is available on the current platform.zPUsing 'pin_memory=False' as WSL is detected. This may slow down the performance.FT)rloggerwarningrrrris_pin_memory_availables z Platform.is_pin_memory_availablecCr})z3 Return the memory usage in bytes. r~rrrrget_current_memory_usagesz!Platform.get_current_memory_usagecCr})zA Return the punica wrapper for current platform. r~rrrrget_punica_wrapperrzPlatform.get_punica_wrappercCstdtdfS)zE Return the platform specific values for (-inf, inf) z-infinf)float)rcrirrrget_infinity_valuesszPlatform.get_infinity_valuescCro)zF Checks if the platform allows inplace memory updates Trrrrrcan_update_inplacerzPlatform.can_update_inplacecCro)zK Returns how much padding the LoRA logits need for kernels rrrrrget_lora_vocab_padding_sizerz$Platform.get_lora_vocab_padding_sizecCro)zW Get device specific communicator class for distributed communication. zUvllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBaserrrrrget_device_communicator_clsrz$Platform.get_device_communicator_clscCro)zI Returns whether the current platform supports MX types. Frrrrr supports_mxrzPlatform.supports_mxcCro)zJ Returns whether the current platform supports FP8 types. Frrrrr supports_fp8rzPlatform.supports_fp8cCro)a Returns whether the preferred FP8 type is FNUZ on the current platform. There are two representations of FP8, OCP FP8 and FNUZ FP8. The OCP specification can be found at https://tinyurl.com/b7jvwpft. The FNUZ specification can be found at https://tinyurl.com/5n6hwwu5. AMD's MI300 and MI325 have native hardware support for FNUZ. All other hardware has converged on the OCP FP8 standard. Frrrrr is_fp8_fnuzs zPlatform.is_fp8_fnuzcCrX)z Returns the preferred FP8 type on the current platform. See the documentation for is_fp8_fnuz for details. )rJZ float8_e4m3fnrrrr fp8_dtypeszPlatform.fp8_dtypecCs0ddlm}ddlm}|j}|jp|jdkS)zS Whether to use allgather in LogitsProcessor to gather the logits. rN)get_current_vllm_configZexternal_launcher)Z vllm.envsenvs vllm.configrparallel_configZ VLLM_USE_V1Zdistributed_executor_backend)rcrrrrrruse_all_gathers  zPlatform.use_all_gather model_configcCro)zjReturns whether the current platform can support v1 for the supplied model configuration. Frrcrrrr supports_v1rzPlatform.supports_v1cCs ||S)zN Returns whether the current platform supports v1 by default. )rrrrr default_v1s zPlatform.default_v1cCro)zR Returns if custom allreduce is supported on the current platform Frrrrruse_custom_allreducerzPlatform.use_custom_allreducepromptparamsprocessed_inputscCro)z6Raises if this request is unsupported on this platformNr)rcrrrrrrvalidate_requestszPlatform.validate_requestkeycCs>tt|jd}|durt||rt||Std|j|dS)Nz1Current platform %s does not have '%s' attribute.)getattrrJr@hasattrrr)r6rrrrr __getattr__s zPlatform.__getattr__cCs |j}|jdur ||_|jS)zE Return the global graph pool for the this platform. N) __class__rIZgraph_pool_handle)r6rcrrrget_global_graph_pools  zPlatform.get_global_graph_poolcCr})zO Returns the total number of compute units (CU) on single GPU. r~rrrrr get_cu_countrzPlatform.get_cu_countcCro)zB Get static graph wrapper class for static graph. z=vllm.compilation.base_static_graph.AbstractStaticGraphWrapperrrrrrget_static_graph_wrapper_clsrz%Platform.get_static_graph_wrapper_clsbackend prefix_store group_rank group_sizetimeoutcCstd|)zI Init platform-specific torch distributed process group. z'Unsupported torch distributed backend: ) RuntimeError)rcrrrrrrrr#stateless_init_device_torch_dist_pg&s z,Platform.stateless_init_device_torch_dist_pgcCro)zU Returns if the kv_cache_dtype is supported by the current platform. Fr)rcrjrrris_kv_cache_dtype_supported4rz$Platform.is_kv_cache_dtype_supported)F)rrM)]rrrr"r;r<rArCrDrErFrGlistrHrIrrpropertyrJrirKboolrOrQrRrSrTrUrVr:r[r\r] classmethodrdrrfrpr1rsrrxrzr|rrrrrrrrrrrrrrr+rrtypesZDevicerrrrrrrrrrrrrrrrr rrr rrrrrr rr rrrrrrr=isJ                              r=c@seZdZejZdZdS)UnspecifiedPlatformrBN)rrrr"r*r>r@rrrrr<sr)/rr`rrrYdatetimerrtypingrrrrrnumpyrrJZtorch.distributedr r Z vllm.inputsr r Z vllm.loggerr rrrZvllm.lora.requestrZvllm.pooling_paramsrZvllm.sampling_paramsrZ vllm.utilsrrrrrEnumrr"r+r1r=rrrrrsH       V