o )ie@sHdZddlmZddlmZddlmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZddlZddlZddlmZmZmZdd lmZdd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$ee%Z&e rnddl'm(Z(dZ)dZ*dZ+e r|ddl,m-Z-dede ffddZ.de/de0de0de0fddZ1de e0de e0de0de0d e0f d!d"Z2de e0de e0de0de0d e0f d#d$Z3d%e/de e0d&e0d'e0de0d(e0d e0de e0e e0ffd)d*Z4ed+d,d-Z5Gd.d/d/ee5Z6Gd0d1d1eZ7d2d3Z8d4d5Z9de/d6e:d7e;fd8d9Zd>ed7e>fd?d@Z?dS)AzAttention backend utils) defaultdict)contextmanager) dataclass) accumulate) TYPE_CHECKINGAnyDictListOptionalTupleTypeTypeVarUnionN)AttentionMetadataAttentionMetadataBuilderAttentionState) AttentionType) ModelConfig) init_logger)MultiModalPlaceholderMap)async_tensor_h2dmake_tensor_with_pad)ModelRunnerBasez@ROCm/HIP is not currently supported with encoder/decoder models.)ModelInputForGPUBuilder block_tablescCs,|durdSt|totdd|DS)zM Check if block_tables is None or a dictionary with all None values. NTcss|]}|duVqdSN).0valuerri/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/attention/backends/utils.py 1sz(is_block_tables_empty..) isinstancedictallvalues)rrrr!is_block_tables_empty*s  r' is_prompt query_len context_lensliding_windowcCs"d}|r|durtd||}|S)z2 Compute the start index of slot mapping. rN)max)r(r)r*r+ start_idxrrr!compute_slot_mapping_start_idx4s r. slot_mapping block_table range_start range_end block_sizec Cs>t||D]}|||}||}|||}||qdSr)rangeappend) r/r0r1r2r3iZ block_number block_offsetslotrrr!_compute_slot_mapping_python?s    r9c CsLt|}t||}||}||}||}||9}||7}||dSr)nparrayZarangeextend) r/r0r1r2r3Zblock_table_arrayidxr7Zseq_slot_mapping_arrayrrr!_compute_slot_mapping_numpyIs  r>is_profile_runseq_idseq_lenr-c Cs|r |tg|dStd||}|tg|t||} |} | | } ||} | tkr8t|| | | |dSt|| | | |dS)z Compute slot mapping. Nr)r< PAD_SLOT_IDr,!_COMPUTE_SLOT_MAPPING_NUMPY_NUMELr9r>) r?r/r@rAr*r-r3rZpadding_mask_lenr1r2Znumelr0rrr!compute_slot_mappingVs  rDTAttentionMetadatar)boundc@s`eZdZUeeed<dddZddZdd d efd d Z d e e de e de de fddZ dS)CommonMetadataBuilder _metadata_cls input_builderrcCs"||_|j|_|j|_|j|_dSr)rIrunnerr+r3)selfrIrrr!__init__s zCommonMetadataBuilder.__init__cCs>g|_g|_g|_g|_g|_tt|_d|_d|_ d|_ dS)Nr) r/prefill_seq_lens context_lensr curr_seq_lensrrmultimodal_placeholder_maps num_prefillsnum_prefill_tokensnum_decode_tokens)rKrrr!prepares zCommonMetadataBuilder.prepare inter_dataz,ModelInputForGPUBuilder.InterDataForSeqGroupchunked_prefill_enabledc Csr|j}|j}t|jdd|jD|j|j|j|j|j D]\}}}}} } } |j | |rX|j } | rC| D] \} }|j | |q6|jd7_|j|7_|j |n| dkseJd|| | |j| 7_|j |g}|jr|||}n|s|s|dur| dkr||}n ||| d}|j |t|}t|| | |j}t||j||| ||j|jqdS)NcSsg|]}t|qSr)len)rtrrr! sz8CommonMetadataBuilder._add_seq_group..z+seq_len: {}, context_len: {}, query_len: {}r)r(rzipZseq_idsZ input_tokensZ orig_seq_lensseq_lens query_lensrNZcurr_sliding_window_blocksr5Zmulti_modal_placeholder_mapsitemsrPr<rQrRrMformatrSrOZprefix_cache_hitr'r.r+rDr/r3)rKrUrVr(rr@Z token_lenrAZ curr_seq_lenr)r*Zcurr_sliding_window_blockZmm_mapsmodalityZ placeholdersr0r?r-rrr!_add_seq_groupsb        z$CommonMetadataBuilder._add_seq_groupr\r]cuda_graph_pad_size batch_sizecCs|jjD] }|||jjq|jj}|dk}t|}t|jdd} t|jdd} |j } t t |dd} t t |dd} |rz|j tg||j g||} |jjd|}t|jD]\}}|rn|||dt|f<q^t|j|dd}n t|jdtj|d}|dksJd ||dusJt|jtj||jj}t|tj||jj}t|j tj||jj}t| tj||jj}t| tj||jj}d d |jD}|j did |j!d |d|ddd|j"d| d|d|d|d| d| d|d|d|d|d|S)aBuild attention metadata with on-device tensors. Args: seq_lens: The maybe padded sequence lengths of the input sequences. query_lens: The query lengths of the input sequences. cuda_graph_pad_size: The padding size for cuda graph. -1 if cuda graph is not used. batch_size: The maybe padded batch size. rr)default)initialNTZ non_blocking)paddtypedevicezquery_lens: {}cSsi|] \}}||qSr)Z index_map)rr`Zplaceholder_maprrr! sz/CommonMetadataBuilder.build..rQr/"multi_modal_placeholder_index_mapsenable_kv_scales_calculationrRrSr\seq_lens_tensor max_query_lenmax_prefill_seq_lenmax_decode_seq_lenquery_start_loc seq_start_loccontext_lens_tensorruse_cuda_graphr)#rIZinter_data_listrarVrJrir,rMrOrSlistrr/r<rBrgraph_block_tables enumeraterWtorch from_numpytorintr_rrNZ pin_memorylongint32rPr^rHrQrR)rKr\r]rbrcrUriZuse_captured_graphrnrorprSrqrrZinput_block_tablesr6r0rrsrmZslot_mapping_tensorZquery_start_loc_tensorZseq_start_loc_tensorZplaceholder_index_mapsrrr!builds            zCommonMetadataBuilder.buildN)rIr) __name__ __module__ __qualname__r rE__annotations__rLrTboolrar r{r~rrrr!rGs    7rGc@seZdZdddZedefddZded dfd d Z dded efddZ dd ed e e e ffddZ dd ed dfddZd ddZdefddZde e e ffddZde e e ffddZdS)!CommonAttentionStaterJrcCs||_d|_dS)NF)rJ_is_graph_capturing)rKrJrrr!rL&s zCommonAttentionState.__init__max_batch_sizeccstd|_tj|fttj|jjd|_tj|tj |jjd|_ t |jj j |jjd|_dVd|_|`|` |`dS)NT)rhri)riF)rrxfullrBr|rJri_graph_slot_mappingZonesr}_graph_seq_lensryrvrz_graph_block_tables)rKrrrr! graph_capture*s*z"CommonAttentionState.graph_capturercreturncCs|jsJ||jSr)r __class__rJ)rKrcrrr! graph_clone@s  z CommonAttentionState.graph_cloneFis_encoder_decoder_modelcCs|jsJ|jjjdiddddd|d|jd|ddddd dd |jd|d d d d ddd|jjddddddd|jd|dd}|rn|jjdvsgJd|jjd|j ||d|S)NrQrrRrSr/rkrlTr\rmrnrZZmax_decode_query_lenrorprqrrrsrrtXFORMERS FLASH_ATTNZ ROCM_FLASH[Expected attn_backend name to be either 'XFORMERS','ROCM_FLASH', or 'FLASH_ATTN', but got '')rc attn_metadatar) rrJ attn_backendZ make_metadatarrmax_seq_len_to_capturerget_name+_update_captured_metadata_for_enc_dec_model)rKrcrrrrr!$graph_capture_get_metadata_for_batchDs`        z9CommonAttentionState.graph_capture_get_metadata_for_batchcCsT|j|jj|jjd}|r(|jjdvs!Jd|jjd|j||d|S)N)r/rmrrrr)r input_buffers)r/decode_metadatarmrrJrr/_add_additional_input_buffers_for_enc_dec_model)rKrrrrrr!get_graph_input_buffersgs   z,CommonAttentionState.get_graph_input_buffersNcCsl|dj|jjdd|dj|jjdd|r4|jjdvs,Jd|jjd|||dSdS)NrmTrfr)rrzMExpected attn_backend name to be either 'XFORMERS' or 'FLASH_ATTN', but got 'r)copy_rrmrrJrr(_prepare_input_buffers_for_enc_dec_model)rKrrrrrr!prepare_graph_input_buffers|s$  z0CommonAttentionState.prepare_graph_input_bufferscCsdSrr)rKZ model_inputrrr! begin_forwardsz"CommonAttentionState.begin_forwardcCstjgtjd|_tj||jfdtjd|_tj|fdtjd|_ tj|fdtjd|_ |jj |_ d|_ dS)a{ Updates the attention metadata parameters for CUDA graph capture in an encoder-decoder model. This method modifies attention-related tensors and metadata required for CUDA graph capture in encoder-decoder models. Specifically, it updates the cross-attention and encoder sequence tensors in the AttentionMetadata object. )rhrZrN)rxZtensorr{cudacross_slot_mappingrrJZget_max_block_per_batchcross_block_tablesencoder_seq_lensencoder_seq_lens_tensorrmax_encoder_seq_lennum_encoder_tokens)rKrcrrrr!rs2     z@CommonAttentionState._update_captured_metadata_for_enc_dec_modelrcCs(|jj|d<|jj|d<|jj|d<dS)a Saves additional input buffers specific to the encoder-decoder model from the attention metadata. This method extracts and stores encoder-decoder related input buffers from the `attn_metadata` into the `input_buffers` dictionary. The buffers include encoder sequence lengths, cross-slot mappings, and cross-block tables, which are essential for the encoder-decoder model during CUDA graph replay. rrrN)rrrrrKrrrrr!rs   zDCommonAttentionState._add_additional_input_buffers_for_enc_dec_modelcCsF|dj|jjdd|dj|jjdd|dj|jjdddS)a Populates input buffers with data from the encoder-decoder model's attention metadata. This method fills the input buffers with encoder-decoder specific tensors. It copies data from the `attn_metadata` and keyword arguments (`kwargs`) into corresponding buffers in the `input_buffers` dictionary. The copied data includes attention-related metadata as well as input IDs and positional information for the encoder. rTrfrrN)rrrrrrrrr!rs  z=CommonAttentionState._prepare_input_buffers_for_enc_dec_model)rJr)F)rN)rrrrLrr{rrrrrstrrrrrrrrrrrr!r$sB  &     rcCs|jduo|jduo|jduS)zG All attention metadata required for encoder attention is set. N)rrrrrrr! is_all_encoder_attn_metadata_sets rcCs|jo |jduo |jduS)z All attention metadata required for enc/dec cross-attention is set. Superset of encoder attention required metadata. N)rrrrrrr!is_all_cross_attn_metadata_sets r attn_typercCsn|tjkr|r |j}n|j}|j||jfS|tjkr"|j|j|j fS|tj kr.|j|jdfSt dt |)al The particular choice of sequence-length- and block-table-related attributes which should be extracted from attn_metadata is dependent on the type of attention operation. Decoder attn -> select entirely decoder self-attention-related fields Encoder/decoder cross-attn -> select encoder sequence lengths & cross-attn block-tables fields Encoder attn -> select encoder sequence lengths fields & no block tables Arguments: * attn_metadata: Attention metadata structure associated with attention op * is_prompt: True if prefill, False otherwise * attn_type: encoder attention, decoder self-attention, encoder/decoder cross-attention Returns: * Appropriate sequence-lengths tensor * Appropriate max sequence-length scalar * Appropriate block tables (or None) NzInvalid attention type ) rZDECODERrorprmrENCODER_DECODERrrrENCODERAttributeErrorr)rr(rZ max_seq_lenrrr!get_seq_len_block_table_argss"   rcCs~d}d}d}|tjkr|jdusJ|j}|j}d}n|tjkr1|jdus'J|j}|j}|j}n |j}|j}|j}|||fS)a Calculate the number of prefill and decode tokens for query, key/value based on the attention metadata and the specified attention type. Args: attn_metadata (AttentionMetadata): Attention Metadata object. attn_type (AttentionType): The type of attention being used. Returns: Tuple[int, int, int]: A tuple containing three integers: - The number of prefill query tokens. - The number of prefill key/value tokens. - The number of decode query tokens. Raises: AssertionError: If the number of encoder tokens in `attn_metadata` is `None` when required for the calculations. rN)rrrrrRrS)rrZnum_prefill_query_tokensZnum_decode_query_tokensZnum_prefill_kv_tokensrrr!&get_num_prefill_decode_query_kv_tokens!s&  rc@s:eZdZUeeed<eed<eed<eed<eed<dS)MLADims q_lora_rank kv_lora_rankqk_nope_head_dimqk_rope_head_dim v_head_dimN)rrrr r{rrrrr!rPs  r model_configcCs(|j}tt|dd|j|j|j|jdS)Nr)rrrrr)hf_text_configrgetattrrrrr)rrrrr! get_mla_dimsYs r)@__doc__ collectionsr contextlibr dataclassesr itertoolsrtypingrrrr r r r r rnumpyr:rxZvllm.attentionrrrZ vllm.attention.backends.abstractrZ vllm.configrZ vllm.loggerrZvllm.multimodalrZ vllm.utilsrrrloggerZvllm.worker.model_runner_baserZSTR_NOT_IMPL_ENC_DEC_ROCM_HIPrBrCZvllm.worker.model_runnerrr'rr{r.r9r>rDrErGrrrrtuplerrrrrrrr!s    ,            '%6   4  /