o )i?'@sddlZddlmZddlmZmZmZmZmZddl m Z m Z ddl m Z ddlmZddlmZddlmZddlmZdd lmZmZerOdd lmZneZeeZed Zed Zed ZedZ ee GdddZ!dS)N)field) TYPE_CHECKINGAnyLiteralOptionalget_args)SkipValidationmodel_validator) dataclass)Self)config) init_logger) GiB_bytesget_cpu_memory)ParallelConfig) @)autoZfp8Zfp8_e4m3Zfp8_e5m2Zfp8_inc)rZfloat32)builtinsha256Zsha256_cbor_64bitc@seZdZUdZdZeeed< dZe ed< dZ e ed< dZ e ed < d Z eed < dZeeed < dZeeed < dZeeed< dZeed< dZe ed< d Zeed< dZeeed< dZeeed< dZeed< dZeed< edd dZeeed< edd dZeeed< d Z eed< de!fddZ"d.dd Z#d!d"Z$e%d#d$de&fd%d&Z'd.d'd(Z(d.d)d*Z)d+e*ddfd,d-Z+dS)/ CacheConfigzConfiguration for the KV cache.N block_sizeg?gpu_memory_utilization swap_spacer cache_dtypeFis_attention_freenum_gpu_blocks_overridesliding_windowenable_prefix_cachingrprefix_caching_hash_algorcpu_offload_gbcalculate_kv_scalescpu_kvcache_space_bytesmamba_page_size_paddedmamba_cache_dtypemamba_ssm_cache_dtype)defaultinitnum_gpu_blocksnum_cpu_blockskv_sharing_fast_prefillreturncCsFg}||j||j||jtjt|dd}|S)a WARNING: Whenever a new field is added to this config, ensure that it is included in the factors list if it affects the computation graph. Provide a hash that uniquely identifies all the configs that affect the structure of the computation graph from input ids/embeddings to the final hidden states, excluding anything before input ids/embeddings and after the final hidden states. F)usedforsecurity) appendrr)r*hashlibmd5strencode hexdigest)selfZfactorsZhash_strr9]/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/config/cache.py compute_hashzs   zCacheConfig.compute_hashcCs |jt|_||dS)N)rrswap_space_bytes_verify_cache_dtype_verify_prefix_cachingr8r9r9r: __post_init__s  zCacheConfig.__post_init__cCsdd|jDS)NcSsi|] \}}|t|qSr9)r5).0keyvaluer9r9r: sz,CacheConfig.metrics_info..)__dict__itemsr?r9r9r: metrics_infoszCacheConfig.metrics_infoafter)modecCsJ|jdkr td|j|jdkrtd|jd|jr#td|S)Nrz0CPU offload space must be non-negative, but got g?z2GPU memory utilization must be less than 1.0. Got .zh--kv-sharing-fast-prefill is currently work in progress and not functional yet (i.e. no prefill savings))r% ValueErrorrr/loggerZ warning_oncer?r9r9r: _verify_argss  zCacheConfig._verify_argscCs:|jdkrdS|jttvrtddStd|j)NrzUsing fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.zUnknown kv cache dtype: )rr CacheDTyperLinforKr?r9r9r:r=s zCacheConfig._verify_cache_dtypecCsZ|jsdS|jdurtjstd|jr)|jttvr+td|jdttddSdS)NzmPrefix caching is not supported with sliding window. Run with --disable-sliding-window to use prefix caching.z'Unknown prefix caching hash algorithm: z. Must be one of rJ) r#r"envsZ VLLM_USE_V1NotImplementedErrorr$rPrefixCachingHashAlgorKr?r9r9r:r>s" z"CacheConfig._verify_prefix_cachingparallel_configcCsjt}|j}|j|}|tdd|tdd}|d|kr%td||d|kr3td|dSdS)Nz.2fz GiB out of the z6 GiB total CPU memory is allocated for the swap space.gffffff?zToo large swap space. g?z!Possibly too large swap space. %s)rZtensor_parallel_sizer<rrKrLwarning)r8rSZtotal_cpu_memoryZnum_gpus_per_nodeZcpu_memory_usagemsgr9r9r:verify_with_parallel_configs     z'CacheConfig.verify_with_parallel_config)r0N),__name__ __module__ __qualname____doc__rr BlockSize__annotations__rfloatrrrNr boolr!rintr"r#r$rRr%r&r'r(r) MambaDTyper*rr-r.r/r5r;r@rGr r rMr=r>rrVr9r9r9r:rsd              r)"r3 dataclassesrtypingrrrrrZpydanticrr Zpydantic.dataclassesr Ztyping_extensionsr Z vllm.envsrPZvllm.config.utilsr Z vllm.loggerr Z vllm.utilsrrZvllm.config.parallelrrWrLr[rNr`rRrr9r9r9r:s*