o )ix@stUddlmZmZddlmZmZddlmZmZddl m Z ddl m Z ddl mZddlmZmZmZmZmZmZmZmZmZddlZdd lmZmZdd lmZm Z m!Z!dd l"m#Z#m$Z$er|ddl%Z%ddl&Z%dd l'm(Z(dd l)m*Z*ddl+m,Z,nede-dZ%edZ.edej/dfZ0ee1d< ee2dej/de2ej/e2dfZ3ee1d< ee2e4ej/dfZ5ee1d< ee0dfZ6ee1d< ee3de7e3e8e9efffZ:ee1d< ee5e7ej/e4fdfZ;ee1d< ee.e2e.fZee1d< e d d!Gd"d#d#Z?ee2d$e2dde7d%fZ@ee1d$< d&e@d'e@d(eAfd)d*ZBee9e@fZCee1d+< e Gd,d-d-ZDe d d!Gd.d/d/eZEe d d!Gd0d1d1eEZFe d d!Gd2d3d3eEZGe d d!Gd4d5d5eEZHGd6d7d7ZIGd8d9d9ee9eDfZJGd:d;d;ZKee9ee?fZLee1d<< Gd=d>d>eZMGd?d@d@eMZNdS)A)ABCabstractmethod)UserDict defaultdict)MappingSequence) dataclass)partial) accumulate) TYPE_CHECKINGAnyLiteralOptional TypedDictTypeVarUnioncastfinalN) NotRequired TypeAlias) LazyLoader full_groupby is_list_of)JSONTreejson_map_leaves)Image) BatchFeature)MultiModalHashDicttorch_Tr torch.Tensor HfImageItem HfVideoItem HfAudioItem ImageItem VideoItem AudioItem ModalityDatac@s:eZdZUdZeeed< eeed< eeed<dS)MultiModalDataBuiltinsz7Type annotations for modality types predefined by vLLM.imageZvideoZaudioN) __name__ __module__ __qualname____doc__r(r%__annotations__r&r'r0r0b/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/multimodal/inputs.pyr)`s    r)F)totalMultiModalDataDictT)frozenc@sXeZdZUdZeed< eed< dZeded< defdd Zd e de fd d Z dS) PlaceholderRangea Placeholder location information for multi-modal data. Example: Prompt: `AAAA BBBB What is in these images?` Images A and B will have: ``` A: PlaceholderRange(offset=0, length=4) B: PlaceholderRange(offset=5, length=4) ``` offsetlengthNr!is_embedreturncCs"|jdur|jSt|jSN)r8r7intsumitemselfr0r0r1get_num_embedss zPlaceholderRange.get_num_embedsothercCsbt||jsdS|j|jf|j|jfksdS|jdur |jduS|jdur*|jduSt|j|jSNF) isinstance __class__r6r7r8nested_tensors_equalr?rAr0r0r1__eq__s     zPlaceholderRange.__eq__) r+r,r-r.r;r/r8rr@objectboolrGr0r0r0r1r5ws r5 NestedTensors)r!.abr9cCst|tjrt|tjot||St|tjr$t|tjo#t||St|tr:t|to9tddt||DSt|trPt|toOtddt||DS||kS)z[Equality check between [`NestedTensors`][vllm.multimodal.inputs.NestedTensors] objects.cs|] \}}t||VqdSr:rE).0a_b_r0r0r1 z'nested_tensors_equal..csrMr:rN)rOrQrPr0r0r1rRrS)rCrTensorequallistallzip)rKrLr0r0r1rEs      rEBatchedTensorInputsc@sLeZdZUdZeed< eed< eed< ded< dedefd d Z d S) MultiModalFieldElemz Represents a keyword argument corresponding to a multi-modal item in [`MultiModalKwargs`][vllm.multimodal.inputs.MultiModalKwargs]. modalitykeydataBaseMultiModalFieldfieldrAr9cCszt||jsdS|jdur|jdu}n|jdur|jdu}nt|j|j}|j|jf|j|jfko<|o.factory)r rZrJ)r?r[r\rer0rcr1_field_factorysz"BaseMultiModalField._field_factoryr]r9cCt)a  Construct [`MultiModalFieldElem`][vllm.multimodal.inputs.MultiModalFieldElem] instances to represent the provided data. This is the inverse of [`reduce_data`][vllm.multimodal.inputs.BaseMultiModalField.reduce_data]. NotImplementedErrorr?r[r\r]r0r0r1 build_elemsszBaseMultiModalField.build_elemsbatch pin_memorycCrgr:rhr?rlrmr0r0r1 _reduce_data$sz BaseMultiModalField._reduce_dataFrmelemscCsHdd|D}tt|dkrtd|dd|D}|j||dS)z Merge the data from multiple instances of [`MultiModalFieldElem`][vllm.multimodal.inputs.MultiModalFieldElem]. This is the inverse of [`build_elems`][vllm.multimodal.inputs.BaseMultiModalField.build_elems]. cSsg|]}t|jqSr0)r`r_rOr=r0r0r1 :z3BaseMultiModalField.reduce_data..rz#Cannot merge different field_types=cSsg|]}|jqSr0rbrOelemr0r0r1rs>rp)lenset ValueErrorro)r?rqrmZ field_typesrlr0r0r1 reduce_data-s  zBaseMultiModalField.reduce_dataN)r+r,r-r.rarfrrJrrZrkrVrIror{r0r0r0r1r^s:   r^c @sHeZdZdZdedededeefddZde ed e defd d Z d S) MultiModalBatchedFieldzo Info: [`MultiModalFieldConfig.batched`][vllm.multimodal.inputs.MultiModalFieldConfig.batched] r[r\r]r9cs |j||dfdd|DS)Nr[r\csg|]}|qSr0r0rr field_factoryr0r1rsPz6MultiModalBatchedField.build_elems..)rfrjr0r~r1rkIsz"MultiModalBatchedField.build_elemsrlrmcst|dkrLt|tjddrLt|dkr|ddS|djtfdd|DrLtjt|g|djR|dj |dj |d}tj ||dS|S) NrrWcheckrc3s|]}|jkVqdSr:shaperu first_shaper0r1rR_z6MultiModalBatchedField._reduce_data..dtypedevicermout) rxrrrT unsqueeze contiguousrrWemptyrrstack)r?rlrmrr0rr1roRs  z#MultiModalBatchedField._reduce_dataN) r+r,r-r.rarJrrZrkrVrIror0r0r0r1r|Bs$  r|c @sreZdZUdZeeeeeefed<dZe ed<de de de dee fd d Z d ee d ede fd dZdS)MultiModalFlatFieldz Info: [`MultiModalFieldConfig.flat`][vllm.multimodal.inputs.MultiModalFieldConfig.flat] [`MultiModalFieldConfig.flat_from_sizes`][vllm.multimodal.inputs.MultiModalFieldConfig.flat_from_sizes] slicesrdimr[r\r]r9csH|j||dt|jtddsttjsJdfdd|jDS)Nr}rWrz,torch.Tensor is required for multiple slicescsg|] }tt|qSr0)rslice)rOsr]rr0r1rs}sz3MultiModalFlatField.build_elems..)rfrrrrCrrTrjr0rr1rkss zMultiModalFlatField.build_elemsrlrmcst|dkrst|tjddrst|dkr|dS|j|jdkt|djdtjffdd |dtfdd |Drs\}}tfd d |D}tj g|||R|dj |dj |d }tj ||j|d S|jdks|Jd dd|DS)NrrWrrtensorcs |jd|jddfSNrr)rrr0r1_shape_before_afters z=MultiModalFlatField._reduce_data.._shape_before_afterc3s|] }|kVqdSr:r0ru)rrr0r1rRsz3MultiModalFlatField._reduce_data..c3s|]}|jVqdSr:rrrrr0r1rRrr)rrz$dim == 0 is required for nested listcSsg|] }|D]}|qqSr0r0)rOrver0r0r1rsz4MultiModalFlatField._reduce_data..) rxrrrTrrrrWr<rrrconcat)r?rlrmZ shape_beforeZ shape_afterZ shape_concatrr0)rrrr1ros"   z MultiModalFlatField._reduce_dataN)r+r,r-r.rrrr/rr;rarJrZrkrVrIror0r0r0r1ris(    rc @sReZdZUdZeed<dedededee fddZ d e ed e defd d Z d S)MultiModalSharedFieldzm Info: [`MultiModalFieldConfig.shared`][vllm.multimodal.inputs.MultiModalFieldConfig.shared] batch_sizer[r\r]r9cCs|j||d}||g|jS)Nr})rfr)r?r[r\r]rr0r0r1rksz!MultiModalSharedField.build_elemsrlrmcCs|dS)Nrr0rnr0r0r1rosz"MultiModalSharedField._reduce_dataN)r+r,r-r.r;r/rarJrrZrkrVrIror0r0r0r1rs&   rc seZdZedefddZe ddedeeeeeefde fddZ e dded d de fd d Z eded e fddZ de deddffdd ZdededeefddZZS)MultiModalFieldConfigr[cCstt|dS)a Defines a field where an element in the batch is obtained by indexing into the first dimension of the underlying data. Args: modality: The modality of the multi-modal item that uses this keyword argument. Example: ``` Input: Data: [[AAAA] [BBBB] [CCCC]] Output: Element 1: [AAAA] Element 2: [BBBB] Element 3: [CCCC] ``` r_r[)rr|r[r0r0r1batchedszMultiModalFieldConfig.batchedrrrcCstt||d|dS)a Defines a field where an element in the batch is obtained by slicing along the first dimension of the underlying data. Args: modality: The modality of the multi-modal item that uses this keyword argument. slices: For each multi-modal item, a slice (dim=0) or a tuple of slices (dim>0) that is used to extract the data corresponding to it. dim: The dimension to extract data, default to 0. Example: ``` Given: slices: [slice(0, 3), slice(3, 7), slice(7, 9)] Input: Data: [AAABBBBCC] Output: Element 1: [AAA] Element 2: [BBBB] Element 3: [CC] ``` ``` Given: slices: [ (slice(None), slice(0, 3)), (slice(None), slice(3, 7)), (slice(None), slice(7, 9))] dim: 1 Input: Data: [[A],[A],[A],[B],[B],[B],[B],[C],[C]] Output: Element 1: [[A],[A],[A]] Element 2: [[B],[B],[B],[B]] Element 3: [[C],[C]] ``` )rrr)rr)r[rrr0r0r1flats0 zMultiModalFieldConfig.flat size_per_itemr!csT|jdkr td|jdgt|fddtt|D}tj||dS)aP Defines a field where an element in the batch is obtained by slicing along the first dimension of the underlying data. Args: modality: The modality of the multi-modal item that uses this keyword argument. slices: For each multi-modal item, the size of the slice that is used to extract the data corresponding to it. dim: The dimension to slice, default to 0. Example: ``` Given: size_per_item: [3, 4, 2] Input: Data: [AAABBBBCC] Output: Element 1: [AAA] Element 2: [BBBB] Element 3: [CC] ``` ``` Given: slices: [3, 4, 2] dim: 1 Input: Data: [[A],[A],[A],[B],[B],[B],[B],[C],[C]] Output: Element 1: [[A],[A],[A]] Element 2: [[B],[B],[B],[B]] Element 3: [[C],[C]] ``` Info: [`MultiModalFieldConfig.flat`][vllm.multimodal.inputs.MultiModalFieldConfig.flat] rz7size_per_item should be a 1-D tensor, but found shape: rcs6g|]}tdddft||dfqSr)r)rOirZ slice_idxsr0r1rsCs z9MultiModalFieldConfig.flat_from_sizes..r)ndimrzrr rangerxrr)r[rrrr0rr1flat_from_sizess 0  z%MultiModalFieldConfig.flat_from_sizesrcCstt||dS)a Defines a field where an element in the batch is obtained by taking the entirety of the underlying data. This means that the data is the same for each element in the batch. Args: modality: The modality of the multi-modal item that uses this keyword argument. batch_size: The number of multi-modal items which share this data. Example: ``` Given: batch_size: 4 Input: Data: [XYZ] Output: Element 1: [XYZ] Element 2: [XYZ] Element 3: [XYZ] Element 4: [XYZ] ``` r)rr)r[rr0r0r1sharedIszMultiModalFieldConfig.sharedr_r9Ncst||_||_dSr:)super__init__r_r[)r?r_r[rDr0r1rks  zMultiModalFieldConfig.__init__r\rlcCs|j|j||Sr:)r_rkr[)r?r\rlr0r0r1rkqsz!MultiModalFieldConfig.build_elems)r)r+r,r- staticmethodrarrrrr;rrrr^rrJrZrk __classcell__r0r0rr1rs6 4 :!rcseZdZdZedefddZedeefddZ ifde eefd d ffd d Z e d efd dZ d e eeffddZZS)MultiModalKwargsItemz A collection of [`MultiModalFieldElem`][vllm.multimodal.inputs.MultiModalFieldElem] corresponding to a data item in [`MultiModalDataItems`][vllm.multimodal.parse.MultiModalDataItems]. r[cCs&t|dtdtdd}t|gS)zConvenience class for testing.dummyr)r[r\r]r_)rZrrrr from_elems)r[Zmm_elemr0r0r1rs zMultiModalKwargsItem.dummyrqcCstdd|DS)NcSsi|]}|j|qSr0r\rur0r0r1 rz3MultiModalKwargsItem.from_elems..)r)rqr0r0r1rszMultiModalKwargsItem.from_elemsr]r9NcsLt|dd|jD}t|dksJd|tt||_dS)NcSsh|]}|jqSr0rrur0r0r1 rwz0MultiModalKwargsItem.__init__..rzFound different modalities=)rrr]valuesrxnextiter _modality)r?r] modalitiesrr0r1rs zMultiModalKwargsItem.__init__cC|jSr:)rr>r0r0r1r[szMultiModalKwargsItem.modalitycCsdd|DS)NcSsi|]\}}||jqSr0rb)rOr\rvr0r0r1rsz1MultiModalKwargsItem.get_data..)itemsr>r0r0r1get_dataszMultiModalKwargsItem.get_data)r+r,r-r.rrarrrZrrrpropertyr[rJrrr0r0rr1rys "rcseZdZdZedddeeeffddZd;de e d d ffd d Z e d dZ e dMultiModalKwargsa A dictionary that represents the keyword arguments to [`torch.nn.Module.forward`][]. The metadata `items` enables us to obtain the keyword arguments corresponding to each data item in [`MultiModalDataItems`][vllm.multimodal.parse.MultiModalDataItems], via [`get_item`][vllm.multimodal.inputs.MultiModalKwargs.get_item] and [`get_items`][vllm.multimodal.inputs.MultiModalKwargs.get_items]. hf_inputsr config_by_keyc s.ttttfttttft}|D]%\}}||}|dur<|||}t |dkr<||<||j  |qt t }|D]L\}} fdd| D} dd| D} t t| dkrptd|d| tt| } t| D]fdd | D}|t |q|qFt|S) Nrcsi|]}||qSr0r0)rOk) elems_by_keyr0r1rrtz3MultiModalKwargs.from_hf_inputs..cSsi|] \}}|t|qSr0)rx)rOrvr0r0r1rrrz0Cannot merge different batch sizes for modality=z! Found: batch_sizes=csg|]}|qSr0r0)rOr)item_idxr0r1rsrz3MultiModalKwargs.from_hf_inputs..)dictrarrZrryrgetrkrxr[addrVrrrzrrrappendrr) rrZkeys_by_modalityr\configrlrqrr[keysZelems_in_modalityZ batch_sizesrr0)rrr1from_hf_inputss4     zMultiModalKwargs.from_hf_inputsr0rr9Ncs.tt|ddd}t||_d|_dS)NcSrr:rxr0r0r1sz+MultiModalKwargs.__init__..r)rrrr_items_by_modality_data)r?rZitems_by_modalityrr0r1rs   zMultiModalKwargs.__init__cCs |jSr:)rrr>r0r0r1rs zMultiModalKwargs.modalitiesFnested_tensorsrmcst|tjr|St|tjrt|St|ttfrt|Sfdd|D}t |tjdds2|St t tj|t dkrId dStfddDrVStjt gdjRdjdjd }tj|d S) z Stack the inner dimensions that have the same shape in a nested list of tensors. Thus, a dimension represented by a list means that the inner dimensions are different for each element along that dimension. csg|]}t|qSr0r _try_stackrOtrpr0r1rss z/MultiModalKwargs._try_stack..rWrrrc3s |] }|jdjkVqdS)rNrr)tensors_r0r1rRsz.MultiModalKwargs._try_stack..rr)rCrrTnpndarrayZ from_numpyr;floatrrrrVrxrranyrrrrr)rrmZstackedZoutputsr0)rmrr1rs0       zMultiModalKwargs._try_stack inputs_listcsdt|dkriSttttft}|D]}|D] \}}|||qqfdd|DS)a Batch multiple inputs together into a dictionary. The resulting dictionary has the same keys as the inputs. If the corresponding value from each input is a tensor and they all share the same shape, the output value is a single batched tensor; otherwise, the output value is a list containing the original value from each input. rcsi|] \}}|t|qSr0r)rOrZ item_listrpr0r1rs z*MultiModalKwargs.batch..)rxrrarVrJrr)rrmZ item_listsZinputsrrr0rpr1rls  zMultiModalKwargs.batchbatched_inputsrcs,tttj|}tfdd|}tt|S)Ncs|jddS)NT)rZ non_blocking)torrr0r1r*sz,MultiModalKwargs.as_kwargs..)rrrrTrrY)rrZ json_inputsZ json_mappedr0rr1 as_kwargs!s   zMultiModalKwargs.as_kwargscC |Sr:)rrr>r0r0r1r0 zMultiModalKwargs.keyscCrr:)rrr>r0r0r1r3rzMultiModalKwargs.valuescCrr:)rrr>r0r0r1r6rzMultiModalKwargs.itemsr\cCs|||Sr:)rr)r?r\defaultr0r0r1r9szMultiModalKwargs.getcOs`t|}|j|g|Ri|}|jD]}|D]}|j|g|Ri|qqd|_|Sr:)rrpoprrr)r?r\argskwargsr]resrr=r0r0r1r<s zMultiModalKwargs.popcCs t|Sr:)rrr>r0r0r1__iter__HrzMultiModalKwargs.__iter__cCs ||Sr:)r)r?r\r0r0r1 __getitem__KrzMultiModalKwargs.__getitem__rAcCst||jsdS|j|jkSrB)rCrDrrFr0r0r1rGNs  zMultiModalKwargs.__eq__ method_namer[cCsF|js td|d||jvr!t|j}td|d|dS)N`zH` is not supported when MultiModalKwargs is not initialized with `items`z Modality z" not found. Available modalities: )r RuntimeErrorryrKeyError)r?rr[Zavailable_modalitiesr0r0r1_validate_modalityTs   z#MultiModalKwargs._validate_modalitycCs|d|t|j|S)z0Get the number of items belonging to a modality.get_item_count)rrxrr?r[r0r0r1r_s zMultiModalKwargs.get_item_count item_indexcCs|d||j||S)zr Get the keyword arguments corresponding to an item identified by its modality and index. get_itemrr)r?r[rr0r0r1rds zMultiModalKwargs.get_itemcCs|d||j|S)zg Get the keyword arguments corresponding to each item belonging to a modality. get_itemsrrr0r0r1rls  zMultiModalKwargs.get_itemsrpcs~|jdur|jSttttft}|jD]}|D]}|D] \}}|||q!qqfdd|D}||_|S)Ncs2i|]\}}t|dkr||djj|dqS)rrp)rxr_r{)rOr\rqrpr0r1rsz-MultiModalKwargs.get_data..) rrrarVrZrrrr)r?rmrrr=r\rvr]r0rpr1rts  zMultiModalKwargs.get_data)r0)Fr:)(r+r,r-r.rrrarrrrrrrrJrIrrVrYrlrtypesZDevicerrrrrrrrrHrGrr;rrrrrr0r0rr1rsf   !  +       rMultiModalPlaceholderDictc@sxeZdZUdZeded< eed< eeed< e eeed< e ed< e ded < d ed < e eed <d S)MultiModalInputsz Represents the outputs of [`BaseMultiModalProcessor`][vllm.multimodal.processing.BaseMultiModalProcessor], ready to be passed to vLLM internals. Z multimodalr`promptZprompt_token_idsZtoken_type_idsZ mm_kwargsrZ mm_hashesrZmm_placeholdersZ cache_saltN) r+r,r-r.r r/rarVr;rrrr0r0r0r1rs$     rc@s:eZdZUdZeed< eeed< eeeed<dS)MultiModalEncDecInputsz Represents the outputs of [`EncDecMultiModalProcessor`][vllm.multimodal.processing.EncDecMultiModalProcessor] ready to be passed to vLLM internals. Zencoder_promptZencoder_prompt_token_idsZencoder_token_type_idsN) r+r,r-r.rar/rVr;rr0r0r0r1rs  r)Oabcrr collectionsrrcollections.abcrr dataclassesr functoolsr itertoolsr typingr r r rrrrrrnumpyrZtyping_extensionsrrZ vllm.utilsrrrZvllm.utils.jsontreerrrZ torch.typesZ PIL.ImagerZ%transformers.feature_extraction_utilsrZhasherrglobalsr rr"r/rVr#rr$r%tuplerrar&r'r(r)r3r5rJrIrErYrZr^r|rrrrrrrrr0r0r0r1s   ,      03D&6@&j%