o )i@sddlZddlZddlmZddlmZmZddlmZddl m Z ddl m Z ddl mZmZer9dd lmZndZe eZGd d d ejZGd d d eZdS)N) lru_cache) TYPE_CHECKINGOptional)envs) init_logger)DEFAULT_MAX_NUM_BATCHED_TOKENS)Platform PlatformEnum) VllmConfigc@seZdZdZdZdS)NeuronFrameworkztransformers-neuronxzneuronx-distributed-inferenceN)__name__ __module__ __qualname__TRANSFORMERS_NEURONXNEURONX_DISTRIBUTED_INFERENCErra/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/platforms/neuron.pyr sr c@s&eZdZUejZdZeed<dZ eed<dZ eed<ddgZ e eed<d Z eed <d Zeed <ed)dedefddZedeedefddZededdfddZedefddZedefddZedefddZeedefdd Zeedefd!d"Zd#d$Zd%d&Zd'd(ZdS)*NeuronPlatformneuron device_name device_typeZ neuron_coresray_device_keyZ neuron_quantZ fbgemm_fp8supported_quantizationZgloo dist_backendZNEURON_RT_VISIBLE_CORESdevice_control_env_varr device_idreturncCdS)Nrr)clsrrrrget_device_name$zNeuronPlatform.get_device_name enforce_eagercCr)NFr)rr"rrris_async_output_supported(r!z(NeuronPlatform.is_async_output_supported vllm_configNcCs|j}|jdkr d|_|jdkrd|_|jr|jr|jj|j_|jr>|jjr@t dd|j _ d|j _ t|j jt|j _dSdSdS)Nautoz&vllm.worker.neuron_worker.NeuronWorkerruniz`MLA is enabled on a non-GPU platform; forcing chunked prefill and prefix caching to be disabled.F)parallel_configZ worker_clsZ world_sizeZdistributed_executor_backendZ cache_configZ model_configZ max_model_len block_sizeZuse_mlaloggerinfoZscheduler_configZenable_chunked_prefillZchunked_prefill_enabledmaxrZmax_num_batched_tokens)rr$r'rrrcheck_and_update_config,s(    z&NeuronPlatform.check_and_update_configcCstddS)Nz&Pin memory is not supported on Neuron.F)r)warningrrrris_pin_memory_availableEs z&NeuronPlatform.is_pin_memory_availablecCstjrdStS)NzLvllm.distributed.device_communicators.neuron_communicator.NeuronCommunicator)rZ VLLM_USE_V1r get_device_communicator_clsr.rrrr0Jsz*NeuronPlatform.get_device_communicator_clscCr)NTrr.rrruse_all_gatherQr!zNeuronPlatform.use_all_gathercC0z ddl}W|duStyd}Y|duSwNr)neuronx_distributed_inference ImportError)rr4rrr is_neuronx_distributed_inferenceU  z/NeuronPlatform.is_neuronx_distributed_inferencecCr2r3)transformers_neuronxr5)rr8rrris_transformers_neuronx^r7z&NeuronPlatform.is_transformers_neuronxcCs|s td||}|}tjd}tjj }tj j }||kr*|r*|jS||kr0|s6|dur9|r9tj S|durB|rBtjSdS)aReturn the specified framework if corresponding installations are available. If no framework is specified, use neuronx-distributed-inference by default. If that's unavailable, check and switch to transformers-neuronx. z+Neuron Framework unavailable for platform: ZVLLM_NEURON_FRAMEWORKN) Z is_neuronAssertionErrorr9r6osenvirongetr rvaluer)selfZ tnx_installedZ nxd_installedZspecified_frameworkZ tnx_framework nxd_frameworkrrrget_neuron_framework_to_usegs"     z*NeuronPlatform.get_neuron_framework_to_usecCstj}||kS)a/ Return True if the framework determined in get_neuron_framework_to_use() is NeuronFramework.NEURONX_DISTRIBUTED_INFERENCE, False otherwise. This is used to select the Neuron model framework and framework-specific configuration to apply during model compilation. )r rrA)r?r@rrruse_neuronx_distributeds z&NeuronPlatform.use_neuronx_distributedcCs|tjkS)a& Return True if the framework determined in get_neuron_framework_to_use() is NeuronFramework.TRANSFORMERS_NEURONX, False otherwise. This is used to select the Neuron model framework and framework-specific configuration to apply during model compilation. )rAr r)r?rrruse_transformers_neuronxsz'NeuronPlatform.use_transformers_neuronx)r) r rrr ZNEURONZ_enumrstr__annotations__rrrlistrr classmethodintr rboolr#r r,r/r0r1rr6r9rArBrCrrrrrs:       r)enumr; functoolsrtypingrrZvllmrZ vllm.loggerrZ vllm.utilsrZ interfacer r Z vllm.configr r r)Enumr rrrrrs