o )i@sRddlmZddlmZmZmZddlmZddlm Z e e Z GdddZ dS))Optional)CompilationLevel CUDAGraphMode VllmConfig)BatchDescriptor) init_loggerc@s`eZdZdZdefddZdedefddZd ed e fd d Z ded e ee effddZ dS)CudagraphDispatchera Runtime cudagraph dispatcher to dispach keys for multiple set of cudagraphs. The dispatcher stores two sets of dispatch keys, one for PIECEWISE and one for FULL cudagraph runtime mode. The keys are initialized depending on attention support and what cudagraph mode is set in CompilationConfig. The keys stored in dispatcher are the only source of truth for valid cudagraphs that can be dispatched at runtime. At runtime, the dispatch method generates the runtime cudagraph mode (FULL, PIECEWISE, or NONE for no cudagraph) and the valid key (batch descriptor) based on the input key. After dispatching (commuicate via forward context), the cudagraph wrappers will trust the dispatch key to do either capturing or replaying (if mode matched), or pass through to the underlying runnable without cudagraph (if mode no match or mode is NONE). vllm_configcCs~||_|j|_|jj|_tjttjti|_|jr:|jj t jkr(|j s:Jd|jd|jj d|jj d|_ dS)NzxCompilation level should be CompilationLevel.PIECEWISE when cudagraph_mode piecewise cudagraphs is used, cudagraph_mode=z, compilation_level=z, splitting_ops=F)r compilation_configcudagraph_moder PIECEWISEsetFULLcudagraph_keysZrequires_piecewise_compilationlevelrZsplitting_ops_contain_attentionZ splitting_opskeys_initialized)selfr rh/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/v1/cudagraph_dispatcher.py__init__s&   zCudagraphDispatcher.__init__ runtime_modebatch_descriptorcCs2|tjtjfvsJd||j||dS)Nz Invalid cudagraph runtime mode: )rr rradd)rrrrrradd_cudagraph_key4sz%CudagraphDispatcher.add_cudagraph_keyr uniform_decode_query_lencs|tjkr|jjD]}||t|ddq |tjkrH| rH|j j j fdd|jjD}|D] }|tjt|ddq:d|_ dS)NF)Z num_tokensZuniform_decodecs g|] }|kr|kr|qSrr).0xZmax_num_tokensrrr Qs zACudagraphDispatcher.initialize_cudagraph_keys..T)Z mixed_moderNONEr Zcudagraph_capture_sizesrrZ decode_moderZseparate_routiner Zscheduler_configZ max_num_seqsr)rr rbsZ"cudagraph_capture_sizes_for_decoderrrinitialize_cudagraph_keys:s*     z-CudagraphDispatcher.initialize_cudagraph_keysreturncCsx|js tdtjdfS||jtjvrtj|fS|j}||jtjvr*tj|fS||jtjvr7tj|fStjdfS)z Given a batch descriptor, dispatch to a cudagraph mode. A new batch descriptor is returned as we might dispatch a uniform batch to a graph that supports a more general batch (uniform to non-uniform). zJcudagraph dispatching keys are not initialized. No cudagraph will be used.N) rloggerZ warning_oncerrrrZ non_uniformr )rrZnon_uniform_keyrrrdispatch[s      zCudagraphDispatcher.dispatchN)__name__ __module__ __qualname____doc__rrrrrintr!tuplerr$rrrrr s  !rN) typingrZ vllm.configrrrZvllm.forward_contextrZ vllm.loggerrr%r#rrrrrs