o )ik @s.ddlZddlZddlZddlmZddlmZmZddlmZm Z m Z m Z m Z m Z mZddlZddlZddlmZmZddlmZerVddlmZddlmZdd lmZddlmZdd lmZdd l m!Z!dd l"m#Z#dd l$m%Z%ddl&m'Z'e%e(Z)da*eGdddZ+eGdddZ,dej-de.dej-fddZ/de,de+de+fddZ0de1e,de+de1e+fddZ2ed Z3Gd!d"d"ej4Z5Gd#d$d$ej6e e3Z7ej8d%d&Z9d'e:fd(d)Z;eGd*d+d+Ze:ee:ee=ed:e=e7e3de=efd?d@ZD AdTde+dBeAdeEeAeAeAeAffdCdDZF AdTdEdFdGdHdBeAdeGfdIdJZHdKe ej-dfdLeAdfgZId9e:dMe dNe1eEe:e e fde fdOdPZJdMe de fdQdRZKdS)UN)abstractmethod) dataclassmake_dataclass) TYPE_CHECKINGAnyCallableClassVarGenericOptionalTypeVar) VllmConfigget_layers_from_vllm_config)cdiv) AttentionImpl)SchedulerOutput) InputBatch)AttentionBackend) Attention)get_kv_connector_cache_layout) init_logger) AttentionSpecc@seZdZUdZejed<ejed< ejed<ejed< ejed< eed< eed< eed < ejed <ejed <d Ze ed <dS)CommonAttentionMetadataz Per-batch attention metadata, shared across layers and backends. AttentionMetadataBuilder instances use it to construct per-layer metadata. For many of the tensors we keep both GPU and CPU versions. query_start_locquery_start_loc_cpuseq_lens seq_lens_cpunum_computed_tokens_cpunum_reqsnum_actual_tokens max_query_lenblock_table_tensor slot_mappingTcausalN) __name__ __module__ __qualname____doc__torchTensor__annotations__intr"boolr,r,l/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/v1/attention/backends/utils.pyr"s&        rc@seZdZUeed<eed<dS) UbatchSlice request_slice token_sliceN)r#r$r%slicer)r,r,r,r-r.Ds  r.rr/returncCs||j|jd||jS)z Creates a new query_start_loc that corresponds to the requests in request_slice. Note: This function creates a new tensor to hold the new query_start_locs. This will break cudagraph compatibility. )startstop)rr/r,r,r-slice_query_start_locsJs r6 ubatch_slice attn_metadatac Cs|j}|j}t|j|}t|dksJdt|t|j|}|j|}|j|}|j|}|j |j } |j |j } t t t |dd|dd} |j|} |j|} t|||||| | | | | d S)z| This function creates a new CommonAttentionMetadata that corresponds to the requests included in ubatch_slice z3query_start_loc must have at least 2 elements, got r3N) rrrrrrrrr r!)r/r0r6rlenrrrrr5r4r*r'maxabsitemr r!r)r7r8r/r0rrrrr num_requestsrrr r!r,r,r-_make_metadata_with_sliceYsP       r@ ubatch_slicescommon_attn_metadatacCs"g}|D] }|t||q|S)z Creates a new CommonAttentionMetadata instance that corresponds to the requests for each UbatchSlice in ubatch_slices. Note: This function does not modify common_attn_metadata )appendr@)rArBresultsr7r,r,r-split_attn_metadatas  rEMc@s&eZdZdZdZ dZ dZ dZdS)AttentionCGSupportz Constants for the cudagraph support of the attention backend Here we do not consider the cascade attention, as currently it is never cudagraph supported.r9r3rN)r#r$r%r&ZALWAYSZ UNIFORM_BATCHZUNIFORM_SINGLE_TOKEN_DECODENEVERr,r,r,r-rGsrGc@seZdZUejZeeed<dZee e ed<e de de ededejfdd Ze dd e d ed edefddZd edefddZd ede defddZd e dejde de dededede defddZdS) AttentionMetadataBuildercudagraph_supportNreorder_batch_threshold kv_cache_spec layer_names vllm_configdevicecCs ||_dSN)rM)selfrMrNrOrPr,r,r-__init__s z!AttentionMetadataBuilder.__init__Fcommon_prefix_lenrB fast_buildr2cCst)a Central method that builds attention metadata. Some builders (MLA) require reorder_batch to be called prior to build. Args: common_prefix_len: The length of the common prefix of the batch. common_attn_metadata: The common attention metadata. fast_build: The meta-data will prioritize speed of building over then speed at execution. Can be used for spec-decode where the result of a build call may only be used for few layers/iters. )NotImplementedErrorrRrTrBrUr,r,r-buildszAttentionMetadataBuilder.buildcCs|jd|dS)z Build attention metadata for CUDA graph capture. Uses build by default. Subclasses that override this method should call self.build or super().build_for_cudagraph_capture. r)rTrBrX)rRrBr,r,r-build_for_cudagraph_capturesz4AttentionMetadataBuilder.build_for_cudagraph_capture draft_indexcCs|jd|ddS)a Build attention metadata for draft model. Uses build by default. Args: common_attn_metadata: The common attention metadata. draft_index: The index of the current draft operation. When speculating a chain of tokens, this index refers to the draft attempt for the i-th token. For tree-based attention, this index instead refers to the draft attempt for the i-th level in the tree of tokens. rT)rTrBrUrY)rRrBr[r,r,r-build_for_draftingsz+AttentionMetadataBuilder.build_for_drafting query_lensnum_query_heads num_kv_heads use_alibiuse_sliding_windowuse_local_attentionnum_smsc CsdS)NFr,) rRrTr]r^r_r`rarbrcr,r,r-use_cascade_attentions z.AttentionMetadataBuilder.use_cascade_attentionF)r#r$r%rGrIrKrr)rLr r*rrliststrr r'rPrSrr+rFrXrZr\npZndarrayrdr,r,r,r-rJsh        rJcCsDtdurt}td||Stj}|durt}|Std||S)NzM`_KV_CACHE_LAYOUT_OVERRIDE` variable detected. Setting KV cache layout to %s.zT`VLLM_KV_CACHE_LAYOUT` environment variable detected. Setting KV cache layout to %s.)_KV_CACHE_LAYOUT_OVERRIDEloggerZ info_onceenvsZVLLM_KV_CACHE_LAYOUTr cache_layoutr,r,r-get_kv_cache_layoutsrnrmcCs|adSrQ)rirlr,r,r-set_kv_cache_layoutsroc@s:eZdZUdZeed<eeed<eed<dZe ed<dS)PerLayerParametersa Currently, FlashInfer backend only support models in which all layers share the same values for the following hyperparameters. Should not be used for trtllm-gen backend since it supports different values for the following hyperparameters. window_leftlogits_soft_capsm_scaleF has_sinksN) r#r$r%r&r*r)r floatrtr+r,r,r,r-rps  rprOrNcls_rc Cst|t|}i}|D]8\}}|j}t||sJt|dd}|dur(|dnd} t|dd} |j} t|dddu} t| | | | ||<q |S)zc Scan layers in `layer_names` and determine some hyperparameters to use during `plan`. Zsliding_windowNrr:rrZsinks)r ritemsimpl isinstancegetattrscalerp) rOrNrvZlayersper_layer_paramskeylayerrxZ window_sizerqrrrsrtr,r,r-get_per_layer_parameters#s    rr|cCs`t|dks Jdt|}|d}tjs.|D]}|j|jkr%td||ks-Jdq|S)ae Currently, FlashInfer backend other than trtllm-gen only support models in which all layers share the same values for the following hyperparameters: - `window_left` - `logits_soft_cap` - `sm_scale` So this function asserts that all layers share the same values for these hyperparameters and returns the global values. rz'No attention layers found in the model.zcWindow left is not the same for all layers. One potential fix is to set disable_sliding_window=TruezFlashInfer backend currently only supports models in which alllayers share the same values for the following hyperparameters:`window_left`, `logits_soft_cap`, `sm_scale`.)r;rfvaluesrkZVLLM_USE_TRTLLM_ATTENTIONrq ValueError)r|Z param_setsZ global_paramsparamsr,r,r-infer_global_hyperparameters@s   rattn_chunk_size block_sizec Csr|j}|j}|j}|jj}|dd|dd}|jd}t||||| tj } ||| } dt || |} t | } | d} t | | | }tj| tj d|}t | | |d}t || | }| ||dk<t|||d||dk||dk<tt |d tj }tj| d|tj d}| || d<||}t || ||t | | }||}||dksJd|d|||}ttj|tj d| |ftj|dd}|j|jddd }t tj|tj d| |}|||f| d}t|}t|}t||j|d d ||j|d d t|t||j|||jd d S) Nr3r:r)Zdtype)r3rzattn_chunk_size z is not divisible by block_size )Zaxis)r<T)rPZ non_blocking) rrrrrrrrr r!r")rnumpyrr rrPshaperhminimumZastypeZint32rZcumsumrepeatarangepadfullZ broadcast_toZ expand_dimsflattenZclipviewr'Z from_numpyrtor;rr<r!)rrBrZquery_start_loc_npZ seq_lens_npZ block_tablerPZ q_seqlensZactual_batch_sizeZq_tokens_in_first_blockZtokens_in_last_blockZ local_blocksZ cu_num_blocksZvirtual_batchesZ block_offsetsrZrarangeZseqlens_q_localZcu_seqlens_q_localZseqlens_k_localZnum_computed_tokens_localZk_seqstarts_absoluteZ block_startsZpages_per_local_batchZ block_indicesZ batch_indicesZblock_table_localrrr,r,r-$make_local_attention_virtual_batchess                 r name_prefix builder_clsbuild_preprocess_fncs@|j} d dtdtdtffdd }t|fd|i}|S) z Return a new subclass of `builder_cls` whose .build(...) method first calls build_preprocess_fn(common_attn_metadata) on the metadata. FrTrBrUcs||||SrQrYrWrrr,r-rX(sz2subclass_attention_metadata_builder..buildrXNre)r#r*rr+type)rrrnamerXWrappedr,rr-#subclass_attention_metadata_builders rattention_backend_clscs$||j}t||fdfddiS)zN Return a new subclass where `get_builder_cls` returns `builder_cls`. Zget_builder_clscsSrQr,r,rr,r-Csz,subclass_attention_backend..)r#r)rrrrr,rr-subclass_attention_backend9s rr3decode_thresholdc Cs|j}|j}|j}|j}||kr|d|dfS|dd|dd}||k}t|s1|d|dfS|jdd}t ||d|ksHJt |d||ksUJ|} || } ||} || } | | | | fS)a, Assuming a reordered batch, finds the boundary between prefill and decode requests. Args: common_attn_metadata: CommonAttentionMetadata object containing the batch metadata. decode_threshold: The maximum query length to be considered a decode. Returns: num_decodes: The number of decode requests. num_prefills: The number of prefill requests. num_decode_tokens: The number of tokens in the decode requests. num_prefill_tokens: The number of tokens in the prefill requests. rr3Nr:)dim) rrrrr'anyr*Zargmaxr>all) rBrrr num_tokensrr]Z is_prefillZ first_prefill num_decodes num_prefillsnum_decode_tokensnum_prefill_tokensr,r,r-split_decodes_and_prefillsFs$     r input_batchrscheduler_outputrcCsg}g}d}d}t|jD] \}}|j|} | |kr$|||| 7}q |||| 7}q t|} t|} d} tdt| | dD]}|| |} | | krQ| S|||d| d} qB| S)z Reorders the batch to split into prefill and decode requests; places all requests with <= decode_threshold tokens at the front of the batch. Returns: True if the batch was modified, False otherwise. rFr3T) enumerateZreq_idsZnum_scheduled_tokensrCr;rangeminZ swap_states)rrrZdecodesZprefillsrriZreq_idrrrZmodified_batchZ decode_idxr,r,r-+reorder_batch_to_split_decodes_and_prefillsps,       rZlogits_indices_paddedZnum_logits_indices metadata_clsfieldscCs||j}t|||fd}|S)zH Return a new subclass of `metadata_cls` with additional fields )bases)r#r)rrrrrr,r,r-subclass_attention_metadatas rcCstd|tdS)zB Return a new subclass of `metadata_cls` for fast prefill ZKVSharingFastPrefill)rrr)r'KV_SHARING_FAST_PREFILL_METADATA_FIELDS)rr,r,r-/make_kv_sharing_fast_prefill_attention_metadatas r)r)r3)Labcenum functoolsr dataclassesrrtypingrrrrr r r rrhr'Z vllm.configr r Z vllm.utilsrZ vllm.attention.backends.abstractrZvllm.v1.core.sched.outputrZvllm.v1.worker.gpu_input_batchrZ vllm.envsrkrZvllm.attention.layerrZ/vllm.distributed.kv_transfer.kv_connector.utilsrZ vllm.loggerrZvllm.v1.kv_cache_interfacerr#rjrirr.r(r1r6r@rfrErFEnumrGABCrJ lru_cachernrgrorprdictrrr*rrrtuplerr+rrrrr,r,r,r-s $          !  / L     Z       - @