o )i1@sxddlmZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZmZdd lmZd d lmZmZmZd d lmZmZmZmZer^dd l m!Z!ee"Z#e de$e j%dZ&e dedZ'e deddZ(Gddde e(Z)Gddde e'Z*Gddde e'Z+eddGdddee'Z,ed dde-de.fd d!Z/Gd"d#d#Z0dS)$)Mapping) dataclass) lru_cache) TYPE_CHECKINGGenericOptionalProtocolTypeVarN)InputProcessingContext) init_logger) AnyTokenizercached_tokenizer_from_config) ClassRegistry)BaseMultiModalProcessorBaseProcessingInfoProcessingCache)BaseDummyInputsBuilderDummyDecoderDataDummyEncoderDataMultiModalProfiler) ModelConfigN)bound_I_I_coT)r covariantc@s"eZdZdZdedefddZdS)ProcessingInfoFactory Constructs a [`BaseMultiModalProcessor`][vllm.multimodal.processing.BaseMultiModalProcessor] instance from the context. ctxreturncCdSN)selfrr#r#d/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/multimodal/registry.py__call__&szProcessingInfoFactory.__call__N)__name__ __module__ __qualname____doc__r rr&r#r#r#r%rsrc@s&eZdZdZdedeefddZdS)DummyInputsBuilderFactoryz Constructs a [`BaseDummyInputsBuilder`][vllm.multimodal.profiling.BaseDummyInputsBuilder] instance from the context. infor cCr!r"r#)r$r,r#r#r%r&4sz"DummyInputsBuilderFactory.__call__N)r'r(r)r*rrr&r#r#r#r%r+-sr+c @s<eZdZdZdddedeedeedeefdd Z dS) MultiModalProcessorFactoryrNcacher, dummy_inputsr/r cCr!r"r#)r$r,r0r/r#r#r%r&?sz#MultiModalProcessorFactory.__call__) r'r(r)r*rrrrrr&r#r#r#r%r-8s r-)frozenc@sNeZdZUeeed<eeed<eeed<dddede e fdd Z dS) _ProcessorFactoriesr, processorr0Nr.rr/cCs$||}||}|j|||dS)Nr.r,r0r3)r$rr/r,Zdummy_inputs_builderr#r#r%build_processorOs  z#_ProcessorFactories.build_processor) r'r(r)rr__annotations__r-r+r rrr5r#r#r#r%r2Is   r2)maxsizemodel_id capacity_gbcCs|dkrt|SdS)Nr)r)r8r9r#r#r%_get_processor_cache\sr:c @seZdZdZd1ddZd2dd Zdddefd d Zdddefd d ZdddefddZ ddde e e ffddZ ddde e e ffddZddde e e ffddZddde fddZddde e e ffddZdeedeedeefddZd2dd Z d3ddd!eedefd"d#Zdd$ddd!eedefd%d&Zddd'ddd!eed(eedeefd)d*Z d3ddd+e d,ee e e fde fd-d.Z! d3ddd+e d,ee e e fde"fd/d0Z#dS)4MultiModalRegistryzL A registry that dispatches data processing according to the model. r NcCsttjtf|_dSr")rnnModuler2_processor_factories)r$r#r#r%__init__fszMultiModalRegistry.__init__ model_configrcCs|j}|j}t||Sr")modelmm_processor_cache_gbr:)r$r@r8r9r#r#r%r:js z'MultiModalRegistry._get_processor_cachecCs||}r |dS)z'Reset the multi-modal processing cache.T)r:reset)r$r@Zprocessor_cacher#r#r%reset_processor_cacheosz(MultiModalRegistry.reset_processor_cachecCs ||sdS|}|jdkS)zWhether the multi-modal input cache should be enabled. NOTE: This is put under MultiModalRegistry on purpose to respect text-only mode for multimodal models. Fr)supports_multimodal_inputsget_multimodal_configrB)r$r@ mm_configr#r#r%enable_mm_input_cachevs  z(MultiModalRegistry.enable_mm_input_cachecsP|jsdS|j|dd}|}|tfdd|Dr&tddSdS)z Checks if the model supports multimodal inputs. Returns True if the model is multimodal with any non-zero supported modalities, otherwise returns False, effectively running in text-only mode. FN tokenizerc3s|] }|dkVqdS)rN)Zget_limit_per_prompt).0modalityrGr#r% s  z@MultiModalRegistry.supports_multimodal_inputs..zcAll limits of multimodal modalities supported by the model are set to 0, running in text-only mode.T)is_multimodal_model_create_processing_infoZget_supported_mm_limitsrFallloggerZ info_once)r$r@r,Zsupported_modalitiesr#rMr%rEs z-MultiModalRegistry.supports_multimodal_inputscCsJ|jsiS|j|dd}t|}|j}||}||dd|DS)z Get the maximum number of tokens per data item from each modality based on underlying model configuration. F disable_cachecSsi|] \}}|dkr|dqS)rrr#)rKrLlimitr#r#r% szJMultiModalRegistry.get_max_tokens_per_item_by_modality..)rOcreate_processorrZ max_model_lenget_mm_limits_per_promptZget_mm_max_contiguous_tokensitems)r$r@r3profilerseq_len mm_limitsr#r#r%#get_max_tokens_per_item_by_modalitys z6MultiModalRegistry.get_max_tokens_per_item_by_modalityc&||fdd||DS)aW Get the maximum number of tokens per data item from each modality based on underlying model configuration, excluding modalities that user explicitly disabled via `limit_mm_per_prompt`. Note: This is currently directly used only in V1 for profiling the memory usage of a model. cs"i|] \}}|dkr||qS)rr#rKkeyZmax_tokens_per_mm_itemr\r#r%rVs  zRMultiModalRegistry.get_max_tokens_per_item_by_nonzero_modality..rXr]rYr$r@r#rar%+get_max_tokens_per_item_by_nonzero_modalitys  z>MultiModalRegistry.get_max_tokens_per_item_by_nonzero_modalitycr^)zx Get the maximum number of tokens from each modality for profiling the memory usage of a model. csi|] \}}|||qSr#r#r_rar#r%rVs zAMultiModalRegistry.get_max_tokens_by_modality..rbrcr#rar%get_max_tokens_by_modalitys   z-MultiModalRegistry.get_max_tokens_by_modalitycCst||S)zq Get the maximum number of multi-modal tokens for profiling the memory usage of a model. )sumrevaluesrcr#r#r%get_max_multimodal_tokenssz,MultiModalRegistry.get_max_multimodal_tokenscCs(|jsiS|j|dd}t|}|S)z Get the maximum number of multi-modal input instances for each modality that are allowed per prompt for a model class. FrS)rOrWrZ get_mm_limits)r$r@r3rZr#r#r%rXs z+MultiModalRegistry.get_mm_limits_per_promptr3r,r0cs dtdtffdd }|S)a/ Register a multi-modal processor to a model class. The processor is constructed lazily, hence a factory method should be passed. When the model receives multi-modal data, the provided function is invoked to transform the data into a dictionary of model inputs. model_clsr cs6jj|ddrtd|tdj|<|S)NT)strictzfModel class %s already has a multi-modal processor registered to %s. It is overwritten by the new one.r4)r>containsrRwarningr2)rir0r,r3r$r#r%wrappers z6MultiModalRegistry.register_processor..wrapper)r)r$r3r,r0rnr#rmr%register_processorsz%MultiModalRegistry.register_processorcCsddlm}||\}}|S)Nr)get_model_architecture)Z vllm.model_executor.model_loaderrp)r$r@rpri_r#r#r%_get_model_clss  z!MultiModalRegistry._get_model_clsrJcCs |dur |js t|}t||Sr")Zskip_tokenizer_initr r )r$r@rJr#r#r%_create_processing_ctxs z)MultiModalRegistry._create_processing_ctxrIcCs*||}|j|}|||}||Sr")rrr>rsr,)r$r@rJri factoriesrr#r#r%rP"s    z*MultiModalRegistry._create_processing_info)rJrTrTcCsf|js t|jd|dur|j }||}|j|}|||}|r'dn||}|j||dS)zT Create a multi-modal processor for a specific model and tokenizer. z is not a multimodal modelNr.) rO ValueErrorrAZenable_mm_processor_cacherrr>rsr:r5)r$r@rJrTrirtrr/r#r#r%rW-s     z#MultiModalRegistry.create_processorr[ mm_countscCsR|j|dd}t|}|||}|j}t||kr'td|dt|d|S) Create dummy data for profiling the memory usage of a model. The model is identified by ``model_config``. FrSzExpected at least z' dummy tokens for profiling, but found z tokens instead.)rWrget_decoder_dummy_dataprompt_token_idslenAssertionErrorr$r@r[rvr3rZZ dummy_dataZ token_idsr#r#r%rxFs   z)MultiModalRegistry.get_decoder_dummy_datacCsJ|j|dd}t|}|||}|j}t||kr#td|t||S)rwFrSzUExpected at least %d dummy encoder tokens for profiling, but found %d tokens instead.)rWrget_encoder_dummy_dataryrzrRZ warning_oncer|r#r#r%r}^s   z)MultiModalRegistry.get_encoder_dummy_data)r N)r@rr")$r'r(r)r*r?r:boolrDrHrErstrintr]rdrerhrXr-rrr+rorrrr r rsrrPrrWrrxrr}r#r#r#r%r;as                  r;)1collections.abcr dataclassesr functoolsrtypingrrrrr Ztorch.nnr<Z vllm.inputsr Z vllm.loggerr Z!vllm.transformers_utils.tokenizerr r Z vllm.utilsr processingrrrZ profilingrrrrZ vllm.configrr'rRtyper=rrrrr+r-r2rrr:r;r#r#r#r%s2