o )iI+@s(ddlmZmZddlmZddlmZmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZdd lmZmZmZmZmZdd lm Z m!Z!m"Z"ee#Z$eGd d d Z%Gd dde Z&Gddde Z'e de!dZ(Gdddee e(Z)Gddde e(Z*dS))ABCabstractmethod)Mapping) dataclassfield)Generic NamedTupleOptionalTypeVarUnioncastN)Image) init_logger)MultiModalDataDictMultiModalEncDecInputsMultiModalInputsMultiModalKwargsMultiModalPlaceholderDict)BaseMultiModalProcessorBaseProcessingInfoEncDecMultiModalProcessorc@sbeZdZUdZeeeefed<e ed<e e dZ e eefed<e e dZe eefed<dS)ProcessorInputszq Represents the keyword arguments to [`vllm.multimodal.processing.BaseMultiModalProcessor.apply`][]. promptmm_data)default_factoryhf_processor_mm_kwargstokenization_kwargsN)__name__ __module__ __qualname____doc__r strlistint__annotations__rrdictrrobjectrr(r(e/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/multimodal/profiling.pyrs rc@seZdZUdZeeed<dS)DummyEncoderDataDummy data used for profiling.prompt_token_idsN)rrr r!r#r$r%r(r(r(r)r*$s r*c@s.eZdZUdZeeed<eed<eed<dS)DummyDecoderDatar+r,multi_modal_datamulti_modal_placeholdersN) rrr r!r#r$r%rrr(r(r(r)r-*s   r-_I)boundc seZdZdZdeddffdd Zedeee fdefdd Z ed e deee fde fd d Z d e deee fde fd dZde de deejfddZde de de deejfddZde de de de deejf ddZZS)BaseDummyInputsBuilderz_ Abstract base class that constructs the dummy data to profile multi-modal models. inforeturnNct||_dSN)super__init__r3)selfr3 __class__r(r)r8;s  zBaseDummyInputsBuilder.__init__ mm_countscCt)zD Build the text input corresponding to `mm_counts`. NotImplementedError)r9r<r(r(r)get_dummy_text@sz%BaseDummyInputsBuilder.get_dummy_textseq_lencCr=)z Build the multimodal input which, after processing, results in the maximum possible number of placeholder tokens. r>r9rAr<r(r(r)get_dummy_mm_dataGs z(BaseDummyInputsBuilder.get_dummy_mm_datacCs,||}|||}ddi}t|||dS)z Build the input which, after processing, results in the maximum possible number of placeholder tokens. Z truncationF)rrr)r@rCr)r9rAr<Z dummy_textZ dummy_mm_datarr(r(r)get_dummy_processor_inputsSs z1BaseDummyInputsBuilder.get_dummy_processor_inputslength num_audioscCs"|dkrgSt|f}|g|S)Nr)npZzeros)r9rErFZaudior(r(r)_get_dummy_audiosds  z(BaseDummyInputsBuilder._get_dummy_audioswidthheight num_imagescCs*|dkrgStjd||fdd}|g|S)NrRGB)color)r new)r9rIrJrKimager(r(r)_get_dummy_imagesos z(BaseDummyInputsBuilder._get_dummy_images num_frames num_videoscCs*|dkrgSt|||dfd}|g|S)NrrM)rGfull)r9rIrJrRrSZvideor(r(r)_get_dummy_videos{s z(BaseDummyInputsBuilder._get_dummy_videos)rrr r!r0r8rrr"r$r@rrCrrDr#nptZNDArrayrHr rQrV __classcell__r(r(r:r)r25s^         r2c sVeZdZdZdeeddffdd ZedefddZ ede efd d Z de e effd d Z dd edee e efdefddZ ddedede e effddZ dd edee e efdefddZ dd edee e efdefddZ  d d edee e efdede e effddZ dd edee e effddZZS)!MultiModalProfilerzL Contains code for running memory profiling for multi-modal models. processorr4Ncr5r6)r7r8rZ)r9rZr:r(r)r8s  zMultiModalProfiler.__init__cC|jjSr6)rZr3r9r(r(r)processing_infoz"MultiModalProfiler.processing_infocCr[r6)rZ dummy_inputsr\r(r(r)r_r^zMultiModalProfiler.dummy_inputscCr[r6)rZZallowed_mm_limitsr\r(r(r) get_mm_limitssz MultiModalProfiler.get_mm_limitsrAr<cCs>|dur|}|j}|||}|jj|j|j|j|jdS)N)rrrr) r`r_rDrZapplyrrrr)r9rAr<factoryZprocessor_inputsr(r(r)_get_dummy_mm_inputssz'MultiModalProfiler._get_dummy_mm_inputsT mm_inputsmm_embeddings_onlycs|d}fdd|DS)Nmm_placeholderscs(i|]\}}|tfdd|DqS)c3s"|] }r |n|jVqdSr6)Zget_num_embedsrE).0itemrer(r) szCMultiModalProfiler._get_mm_num_tokens...)sum)rgZmodalityZ placeholdersrir(r) s z9MultiModalProfiler._get_mm_num_tokens..)items)r9rdreZplaceholders_by_modalityr(rir)_get_mm_num_tokenss z%MultiModalProfiler._get_mm_num_tokensc Cs|||}tt|}|d}t|}tt|j}|jr/t|||}|dg|t|S||krCt j sCt d||t ||t|S)Nencoder_prompt_token_idsraThe encoder sequence length used for profiling (max_num_batched_tokens / max_num_seqs = %d) is too short to hold the multi-modal embeddings in the worst case (%d tokens in total, out of which %s are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.)rcr rlenrrZZpad_dummy_encoder_promptmaxextendenvs VLLM_USE_V1logger warning_oncer"rnr*)r9rAr<rdro total_lenrZZnum_tokens_to_padr(r(r)get_encoder_dummy_datas"     z)MultiModalProfiler.get_encoder_dummy_datac Csv|||}|d}t|}||kr"tjs"td||t||||kr0|dg||t ||d|ddS)Nr,aThe sequence length used for profiling (max_num_batched_tokens / max_num_seqs = %d) is too short to hold the multi-modal embeddings in the worst case (%d tokens in total, out of which %s are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.rZ mm_kwargsrf)r,r.r/) rcrprsrtrurvr"rnrrr-)r9rAr<rdr,rwr(r(r)get_decoder_dummy_datas"   z)MultiModalProfiler.get_decoder_dummy_datacsdur||jj|ddur>durt}ntfdd@D}||kr.zThe sequence length (%d) is smaller than the pre-defined worst-case total number of multimodal tokens (%d). This may cause certain multi-modal inputs to fail during inference. To avoid this, you should increase `max_model_len` or reduce `mm_counts`.ri) r`r]Zget_mm_max_tokens_per_itemrkvalueskeysrurvrcrn)r9rAr<reZtotal_mm_tokensrdr(r{r)_get_mm_max_tokenss2 z%MultiModalProfiler._get_mm_max_tokenscCs|j||ddS)a Returns the maximum length of the multimodal (image placeholders+text) tokens, including any break/text tokens in-between image embeddings. [IMG] [IMG] [IMG] [IMG] [IMG] [IMG] Returns 9, even when the number of image embeddings is 6. This is important to take into account when profiling and initializing the encoder cache size. Fri)r~rBr(r(r)get_mm_max_contiguous_tokens'sz/MultiModalProfiler.get_mm_max_contiguous_tokensr6)T)NT)rrr r!rr0r8propertyrr]r2r_rr"r$r`r rrcboolrnr*rxr-ryr~rrXr(r(r:r)rYsz    $ #  'rY)+abcrrcollections.abcr dataclassesrrtypingrrr r r r numpyrGZ numpy.typingrWZPILr Z vllm.envsrsZ vllm.loggerrZinputsrrrrr processingrrrrrurr*r-r0r2rYr(r(r(r)s&        T