o )i?- @sddlmZddlmZddlmZmZddlZddlm Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZerRdd lmZGdddZeGdddZdededdfddZdejdeejdejfddZ dejdeejdejfddZ! d)de"e#e#fde$ed e"e#ejfd!ee$e$eddf d"d#Z%d e"e#ejfd$e"e#d%fd&e$ejddfd'd(Z&dS)*) defaultdict) dataclass) TYPE_CHECKINGOptionalN)AttentionBackend) ModelConfigSchedulerConfig)MultiModalEmbeddings)extract_layer_index)MultiModalRegistry)AttentionMetadataBuilder)compute_encoder_budget)KVCacheGroupSpec) Attentionc s|eZdZdZdedededededdf fd d Zde e effd d Z defd dZ de dede eeffddZ ZS)MultiModalBudgetzDHelper class to calculate budget information for multi-modal models. model_configscheduler_config mm_registry max_model_len max_num_reqsreturnNcst||_||_||_t|||d\}}||_||_||_||_ | ||_ t t tf}t t tf} ||} | D]\} } || | \} }| || <|| | <q@| |_||_| |_dS)N)rrr)super__init__rrrr max_num_encoder_input_tokensencoder_cache_sizerrZget_mm_limits_per_prompt mm_limitsdictstrintZ+get_max_tokens_per_item_by_nonzero_modalityitems get_max_itemsmax_tokens_by_modality max_items_per_prompt_by_modalitymax_items_per_batch_by_modality)selfrrrrrZencoder_compute_budgetrr"r#r!modality max_tokensmax_items_per_promptmax_items_per_batch __class__`/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/v1/worker/utils.pyrs:      zMultiModalBudget.__init__cCs&|j}t|ddd\}}||fS)NcSs|dS)Nr+)itemr+r+r,Ksz?MultiModalBudget.get_modality_with_max_tokens..)key)r!maxr)r$r!r%r&r+r+r,get_modality_with_max_tokensHs  z-MultiModalBudget.get_modality_with_max_tokenscCst|j|jSN)minrr)r$r+r+r,get_encoder_budgetOsz#MultiModalBudget.get_encoder_budgetr%max_tokens_per_itemc Cs|dkrdS|}|dkrdS||}|j|}tdt||j|}|j}|j}|js5t||j|}||} tdt|| } || fS)Nr)rrr-) r5rr1r4rrrZenable_chunked_prefillZmax_num_batched_tokens) r$r%r6Zencoder_budgetZmax_encoder_items_per_batchZmm_limitr'rrZmax_decoder_items_per_batchr(r+r+r,r Rs0 zMultiModalBudget.get_max_items)__name__ __module__ __qualname____doc__rrr rrtuplerr2r5r __classcell__r+r+r)r,rs0/ rc@s.eZdZUeeed<eed<eeed<dS)AttentionGroupbackendZmetadata_builder layer_namesN) r7r8r9typer__annotations__r listrr+r+r+r,r=s r= mm_embeddingsexpected_num_itemsrcCs|t|tttjfsJdt|dt||ks&Jd|dt|dtdd|Dssz2sanity_check_mm_encoder_outputs..z[Expected multimodal embeddings to be a sequence of 2D tensors, but got tensors with shapes cSsg|]}|jqSr+)shaperGr+r+r, sz3sanity_check_mm_encoder_outputs..N) isinstancerBr;torchTensorr@lenall)rCrDr+r+r,sanity_check_mm_encoder_outputss" rRembedsis_embedcCs8|dur|S|j|jd|jdftjd}|||<|S)a Scatter the multimodal embeddings into a contiguous tensor that represents the placeholder tokens. [`vllm.multimodal.processing.PromptUpdateDetails.is_embed`][]. Args: embeds: The multimodal embeddings. Shape: `(num_embeds, embed_dim)` is_embed: A boolean mask indicating which positions in the placeholder tokens need to be filled with multimodal embeddings. Shape: `(num_placeholders, num_embeds)` Nr)Z fill_value)Znew_fullrKrNnan)rSrT placeholdersr+r+r,scatter_mm_placeholderssrXrWcCs|dur|S||S)z} Reconstructs the embeddings from the placeholder tokens. This is the operation of [scatter_mm_placeholders][]. Nr+)rWrTr+r+r,gather_mm_placeholderss rYshared_kv_cache_layerskv_cache_groups kv_caches attn_groupsc Csi}|r#t|D]\}}t|D]\}}|jD]} ||f|| <qqqnt|D]\}} | jD]} |df|| <q.q'|D]*\} } || || <|| d}||j| |rf|| d}|||j| qdS)aw Bind the allocated KV cache to both ModelRunner and forward context so that the KV cache can be used in the forward pass. This function: 1) Fills the ModelRunner's kv cache list (`runner_kv_caches`) with kv_caches. 2) Associates each attention layer in the `forward_context` with its corresponding KV cache in kv_caches. Args: kv_caches: The allocated kv_caches with layer names as keys. forward_context: The global forward context containing all Attention layers with layer names as keys. runner_kv_caches: The kv_cache declared by ModelRunner. rr-N) rPrrBr r_sortedkeysNotImplementedErrorrkv_cache)r\rbrcZ index2namer`Z layer_indexr?rgr+r+r, bind_kv_caches rhr3)' collectionsr dataclassesrtypingrrrNZ vllm.attention.backends.abstractrZ vllm.configrrZ%vllm.model_executor.models.interfacesr Z vllm.model_executor.models.utilsr Zvllm.multimodal.registryr Z vllm.v1.attention.backends.utilsr Z"vllm.v1.core.encoder_cache_managerr Zvllm.v1.kv_cache_interfacerZvllm.attention.layerrrr=rrRrOrXrYrrrBrarhr+r+r+r,sv          j      5