o )i#@sddlmZddlmZddlZddlmmZddl m Z ddl m Z ddlm Z ddlmZddlmZddlmZmZdd lmZmZdd lmZdd lmZdd lmZd dlmZe Z!edrzddl"m#Z$e%e$drue$ndZ$Wn e&ydZ$YnwdZ$ee'Z(ej)j*j+j,Z-ej)j.j/j,Z0ej)j.j1j,Z2ej)j.j3j,Z4ej)j.j5j,Z6GdddZ7Gddde7Z8Gddde7Z9Gddde7Z:Gddde7Z;Gddde7Ze$durda?d!Z@d"e@e@e@d#e@d#d$ZAe@d#ZB    dMd%ejCd&ejCd'ejCd(eDd)eEd*eEd+eFd,eFd-eFd.eEd/eEd0eFd1eejCd2eejCd3eejCd4eejCd5df"d6d7ZG    dMd%ejCd&ejCd'ejCd(eDd)eEd*eEd+eFd,eFd-eFd.eEd/eEd0eFd1eejCd2eejCd3eejCd4eejCd5df"d8d9ZHed:eGgd;eHejId<ej)j*jJj,ZJGd=d>d>ZKGd?d@d@e7ZLGdAdBdBe7ZMGdCdDdDe7ZNGdEdFdFe7ZOGdGdHdHe7ZPGdIdJdJe7ZQGdKdLdLeZRdS)N) find_spec)OptionalN)auto_functionalized)PatternMatcherPass)enable_symm_mem_for_group) VllmConfig) get_tp_group tensor_model_parallel_all_reduce)get_tensor_model_parallel_rank$get_tensor_model_parallel_world_size) init_logger)current_platform)direct_register_custom_op)VllmInductorPassZ flashinfertrtllm_allreduce_fusionc@s eZdZdejdefddZdS) BasePatterndtypedevicecCs ||_||_t|_t|_dSN)rrrtpr tp_size)selfrrrn/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/compilation/collective_fusion.py__init__.s zBasePattern.__init__N)__name__ __module__ __qualname__torchrstrrrrrrr,src@"eZdZddZdefddZdS)GEMMReduceScatterPatterncCs8tjddg|j|jd}tjddg|j|jd}||gS)Nrrremptyrr)rmul mm_weightrrr get_inputs7sz#GEMMReduceScatterPattern.get_inputspm_passcsPdtjdtjffdd }dtjdtjffdd }t||tj|dS)Nr(r)cs4tjjj||}tjjjj|djjj d}|SNrdim world_size group_name) ropsatenmmdefaultvllmreduce_scatterrr unique_name)r(r)r3r6rrrpattern>s z2GEMMReduceScatterPattern.register..patterncs"tjjj||ddjjjd}|S)Navgr) scatter_dimr0)rr1symm_memZfused_matmul_reduce_scatterr device_groupr0)r(r)gemm_rsr8rr replacementHsz6GEMMReduceScatterPattern.register..replacementrTensorpmregister_replacementr*fwd_onlyrr+r9r?rr8rregister<s   z!GEMMReduceScatterPattern.registerNrrrr*rrFrrrrr"5sr"c@r!)AllGatherGEMMPatterncCs8tjddg|j|jd}tjddg|j|jd}||gSNr$r%r&)rxweightrrrr*YszAllGatherGEMMPattern.get_inputsr+cspdtjdtjdttjtjfffdd }dtjdtjdttjtjfffdd }t||tj|dS)NrJrKreturncs0tjjjj|djjjd}tjjj ||Sr,) rr1r5 all_gatherr4rrr7r2r3)rJrKrMr8rrr9as z.AllGatherGEMMPattern.register..patterncs&tjjj||gdjjjd\}}|S)Nr) gather_dimr0)rr1r<Zfused_all_gather_matmulrr=r0)rJrK ag_output mm_outputsr8rrr?ns z2AllGatherGEMMPattern.register..replacement)rrAtuplerBrCr*rDrErr8rrF_s"  zAllGatherGEMMPattern.registerNrGrrrrrHWsrHc@r!)ScaledMMReduceScatterPatterncCsttjddg|jtd}tjddg|jtddd}tjddg|jtjd}tjddg|jtjd}||||gSNr#r%rr)rr'r FP8_DTYPE contiguous transposefloat32)rinputr)scale_ascale_brrrr*s  z'ScaledMMReduceScatterPattern.get_inputsr+c tdtjdtjdtjdtjdtjf fdd }dtjdtjdtjdtjdtjf fdd }t||tj|dS) NrXmat2rYrZrLc sBtjjjj||||ddjd}tjjjj|djj j d}|S)Nr\rYrZbiasZ scale_result out_dtyperr-) rr1r2 _scaled_mmr4rr5r6rrr7)rXr\rYrZZ scaled_mmr6r8rrr9s  z6ScaledMMReduceScatterPattern.register..patternc s*tjjj||||ddjjjjd}|SNr:r)r;r_r0rr1r<Z"fused_scaled_matmul_reduce_scatterrrr=r0)rXr\rYrZr>r8rrr? z:ScaledMMReduceScatterPattern.register..replacementr@rErr8rrFs"z%ScaledMMReduceScatterPattern.registerNrGrrrrrR}srRc@r!)AllGatherScaledMMPatterncCstjddg|jtd}tjddg|jtddd}|jd|j}tj|dg|jtjd}tjddg|jtjd}||||gSNr#r%rr) rr'rrTrUrVshaperrW)rrJrKs1rYrZrrrr*s  z#AllGatherScaledMMPattern.get_inputsr+c r[) NrJrKrYrZrLc s>tjjjj|djjjd}tjjj j||||ddj dS)Nrr-r]) rr1r5rMr4rrr7r2r`r)rJrKrYrZrMr8rrr9s  z2AllGatherScaledMMPattern.register..patternc s>tjjj||g||gddgdgjgdgjjjd \}}|SNrF)rNZbiasesZ result_scalesZ out_dtypesZuse_fast_accumr0rr1r<Zfused_all_gather_scaled_matmulrrr=r0)rJrKrYrZrOrPr8rrr?  z6AllGatherScaledMMPattern.register..replacementr@rErr8rrFs*z!AllGatherScaledMMPattern.registerNrGrrrrrd rdc@r!)#CutlassScaledMMReduceScatterPatterncCstjddg|jtd}tjddg|jtddd}tjddg|jtjd}tjddg|jtjd}tjddg|j|jd}|||||gSrS)rr'rrTrUrVrWr)rrXr)rYrZcutlass_mm_outputrrrr*s  z.CutlassScaledMMReduceScatterPattern.get_inputsr+c sdtjdtjdtjdtjdtjdtjf fdd }dtjd tjdtjdtjdtjdtjf fd d }t||tj|dS) NrXrKrYrZrnrLc sJtjjjtjjjj|||||dd}tjjjj|ddj j j d}|S)NoutabZa_scalesZb_scalesr^rrr-) rr1 higher_orderr_Ccutlass_scaled_mmr4r5r6rrr7)rXrKrYrZrnrur6r8rrr9s   z=CutlassScaledMMReduceScatterPattern.register..patternr\c s*tjjj||||ddjjjjd}|Srarb)rXr\rYrZrnr>r8rrr? rczACutlassScaledMMReduceScatterPattern.register..replacementr@rErr8rrFs*z,CutlassScaledMMReduceScatterPattern.registerNrGrrrrrmrlrmc@r!)AllGatherCutlassScaledMMPatterncCstjddg|jtd}tjddg|jtddd}|jd|j}tj|dg|jtjd}tjddg|jtjd}|jd}tj||g|j|j d}|||||gSre) rr'rrTrUrVrgrrWr)rrJrKrhrYrZs2outputrrrr* s  z*AllGatherCutlassScaledMMPattern.get_inputsr+c sdtjdtjdtjdtjdtjdtjf fdd }dtjdtjdtjdtjdtjdtjf fd d }t||tj|dS) NrJrKrYrZrxrLc sJtjjjj|djjjd}tjjj tjj j j|||||dd}|dS)Nrr-ror) rr1r5rMr4rrr7rsrrtru)rJrKrYrZrxrMrur8rrr91s  z9AllGatherCutlassScaledMMPattern.register..patternc s>tjjj||g||gddgdgjgdgjjjd \}}|Srirj)rJrKrYrZrxrOrPr8rrr?Hrkz=AllGatherCutlassScaledMMPattern.register..replacementr@rErr8rrF/s2z(AllGatherCutlassScaledMMPattern.registerNrGrrrrrvsrvcsHeZdZdeffdd ZdeedefddZde j fd d Z Z S) AsyncTPPassconfigcst|ttjjtdd|_t|j |j  |jt |j |j  |j|j t jkr]t|j |j  |jt|j |j  |jt|j |j  |jt|j |j  |jdSdS)NZ async_tp_passZ pass_name)superrrrr=r0rpatternsr" model_dtyperrFrHrZbfloat16rRrdrmrv)rrz __class__rrr_s>       zAsyncTPPass.__init__rgrLcCst}|duo ||dkS)Nr)r )rrgrrrris_applicable_for_shapezsz#AsyncTPPass.is_applicable_for_shapegraphcCsD|||d|j|}td|||d|dS)NZbefore_async_tp_passz(Replaced %s patterns with async TP pass.Zafter_async_tp_pass)begin dump_graphr}applyloggerdebug end_and_logrrcountrrr__call__s      zAsyncTPPass.__call__) rrrrrrintboolrfxGraphr __classcell__rrrrry]sryi@)rr$rf allreduce_inresidual rms_gammarms_eps world_rankr/launch_with_pdltrigger_completion_at_endfp32_acc max_token_num pattern_codefuse_rms_quantnorm_out quant_out scale_out scale_factorrLcCs|j\}}|}|||}| ||}|tt|t|k}|rtdus*Jd| dur3|} |}n|}tjdid|d|jdd|d|d| d|d |d |d |d |jd dtd|ddd|d|d| ddd| d|dtj j d|dSt |}|dur|dur| r| durt j j| |||||nCt j j| ||||n7| durt j j|||||} n t j j| ||||dur|durt j j| | ||n t j j| | ||dus| dur||dSdS)Nz0Flashinfer must be enabled when using flashinferrZ token_numrZ residual_in residual_outrrrrr/ hidden_dimZworkspace_ptrsr use_oneshotTrrr allreduce_outrrZ layout_coderr)rg element_sizemin _FI_MAX_SIZESget_DEFAULT_FI_MAX_SIZE_FI_WORKSPACE_TENSORflashinfer_commrZFP4QuantizationSFLayoutZSWIZZLEDr rr1rtZ#fused_add_rms_norm_static_fp8_quantZrms_norm_static_fp8_quantfused_add_rms_normrms_normscaled_fp4_quantstatic_scaled_fp8_quantZcopy_)rrrrrr/rrrrrrrrrrZ num_tokensZ hidden_sizerZcurrent_tensor_sizeZmax_fusion_sizeZuse_flashinferrrrrr call_trtllm_fused_allreduce_norms              rcCsdSrr)rrrrrr/rrrrrrrrrrrrr%call_trtllm_fused_allreduce_norm_fakesr&flashinfer_trtllm_fused_allreduce_norm)rrrrr)Zop_nameZop_funcZ mutates_argsZ fake_impl dispatch_keyc @s>eZdZdZ   ddededededef d d Zd d Zd S)FlashInferFusedAllReduceParamsz5Parameters for FlashInfer fused allreduce operations.Frankr/use_fp32_lamportrrcCs:||_||_||_d|_d|_d|_d|_||_||_dS)NTF) rr/rrrrrrr)rrr/rrrrrrr s z'FlashInferFusedAllReduceParams.__init__cCs"|j|j|j|j|j|j|jdS)N)rr/rrrrr)rr/rrrrrr8rrr!get_trtllm_fused_allreduce_kwargs2sz@FlashInferFusedAllReduceParams.get_trtllm_fused_allreduce_kwargsN)FrF)rrr__doc__rrrrrrrrrs" rcJeZdZdZdedejdedeffdd Z dd Z d e fd d Z Z S) AllReduceRMSNormPatternz This pattern replaces the allreduce + rms norm (without residual) with fused flashinfer implementation. Applies to allreduce + rmsnorm before attn in the first Transformer block. epsilonrrallreduce_paramsct||||_||_dSrr|rrrrrrrrrrrrE z AllReduceRMSNormPattern.__init__cCsPtjgd|j|jd}tjgd|j|jd}tjdg|j|jd}|||gS)Nrrfr$r%r$r&)rrX rms_resultrKrrrr*Ps  z"AllReduceRMSNormPattern.get_inputsr+c\dtjdtjdtjffdd }dtjdtjdtjffdd }t||tj|dS)NrXrrKcs(t|}tt|||jd}|d|fS)NresultrXrKrr)r rRMS_OPr)rXrrKallreduce_outputrmsr8rrr9[s z1AllReduceRMSNormPattern.register..patternc sHt|}ttf|||dd|jtjjdj }|d|dfS)Nrrrrrrrrr) r zeros_likerrrrAllReduceFusionPatternkARResidualRMSNormrr)rXrrKr allreducer8rrr?hs"  z5AllReduceRMSNormPattern.register..replacementr@rErr8rrFYs z AllReduceRMSNormPattern.registerrrrrfloatrrr rrr*rrFrrrrrr>s  rcr) AllReduceFusedAddRMSNormPatternz This pattern replaces the allreduce + rms norm (with residual) with fused flashinfer implementation. Applies to o_proj + rmsnorm after attn and mlp + rmsnorm before attn. rrrrcrrrrrrrrrz(AllReduceFusedAddRMSNormPattern.__init__cCsRtjddg|j|jd}tjddg|j|jd}tjddg|j|jd}|||gSrIr&)rrXrrKrrrr*sz*AllReduceFusedAddRMSNormPattern.get_inputsr+cr)NrrXrKcs,t|}tt|||jd}|d|dfS)NrXrrKrrr)r r RMS_ADD_OPr)rrXrKrrr8rrr9sz9AllReduceFusedAddRMSNormPattern.register..patternc s>ttf||ddd|jtjjdj}|d|dfS)Nrrr)rrrrrrrr)rrXrKrr8rrr?s  z=AllReduceFusedAddRMSNormPattern.register..replacementr@rErr8rrFs z(AllReduceFusedAddRMSNormPattern.registerrrrrrrs  rcBeZdZdZdedejdedeffdd Z de fd d Z Z S) *AllReduceFusedRMSNormStaticQuantFP8Patternz This pattern replaces the allreduce + rms norm (without residual) + static fp8 quant with fused flashinfer implementation. Applies to allreduce + rmsnorm + quant before attn in the first Transformer block. rrrrc&t||||_||_tj|_dSrr|rrrrZ float8_e4m3fn quant_dtyperrrrr z3AllReduceFusedRMSNormStaticQuantFP8Pattern.__init__r+c s~fdd}dtjdtjdtjdtjdtjf fdd }dtjd tjdtjdtjdtjf fd d }t|||tj|dS) Ncstjgdjjd}tjgdjjd}tjgdjjd}tjdgjjd}tjdjtjd}|||||gS)Nrr%r$g?)rZzerosrrr'rZtensorrW)rXrmsnorm_result quant_resultrKscaler8rrr*s   zGAllReduceFusedRMSNormStaticQuantFP8Pattern.register..get_inputsrXrrrKrcs<t|}tt|||jd}tt||d|d}|d|fS)NrrrrXr)r rrrSTATIC_FP8_QUANT_OP)rXrrrKr all_reducermsnorm_out_tuplequant_out_tupler8rrr9s zDAllReduceFusedRMSNormStaticQuantFP8Pattern.register..pattern result_rmsc sJt|}ttf||||d|jtjj|d j }|d|dfS)N rrrrrrrrrr$r) rrrrrrrkARResidualRMSNormFP8Quantrr)rXrrrKrrrr8rrr?s$  zHAllReduceFusedRMSNormStaticQuantFP8Pattern.register..replacementrrArBrCrDrr+r*r9r?rr8rrFs,  z3AllReduceFusedRMSNormStaticQuantFP8Pattern.register rrrrrrrr rrrrFrrrrrr rcr) -AllReduceFusedAddRMSNormStaticQuantFP8Patternz This pattern replaces the allreduce + rms norm (with residual) + static fp8 quant with fused flashinfer implementation. Applies to o_proj + rmsnorm after attn + quant and mlp + rmsnorm + quant before attn. rrrrcrrrrrrrrrz6AllReduceFusedAddRMSNormStaticQuantFP8Pattern.__init__r+c s~fdd}dtjdtjdtjdtjdtjf fdd }dtjdtjdtjdtjdtjf fd d }t|||tj|dS) Ncstjddgjjd}tjddgjjd}tjddgjjd}tjddgjjd}tjddgjtjd}|||||gS)Nr$r%r)rr'rrrrW)rXrrKrrr8rrr* s(   zJAllReduceFusedAddRMSNormStaticQuantFP8Pattern.register..get_inputsrrrXrKrcs@t|}tt|||jd}tt||d|d}|d|dfS)Nrrrr)r rrrr)rrrXrKrrfused_add_rmsnorm_out_tuplerr8rrr96s zGAllReduceFusedAddRMSNormStaticQuantFP8Pattern.register..patternc s@ttf||d|d|jtjj|d j}|d|dfS)Nrr$r)rrrrrrrr)rrrXrKrrr8rrr?Os" zKAllReduceFusedAddRMSNormStaticQuantFP8Pattern.register..replacementrrrr8rrFs,  z6AllReduceFusedAddRMSNormStaticQuantFP8Pattern.registerrrrrrrrrcr) ,AllReduceFusedRMSNormStaticQuantNVFP4Patternz This pattern replaces the allreduce + rms norm (without residual) + static nvfp4 quant with fused flashinfer implementation. Applies to allreduce + rmsnorm + quant before attn in the first Transformer block. rrrrcrrrrrrrro z5AllReduceFusedRMSNormStaticQuantNVFP4Pattern.__init__r+c sfdd}dtjdtjdtjdtjdtjdtjf fd d }dtjd tjdtjdtjdtjdtjf fd d }t|||tj|dS)Ncstjgdjjd}tjgdjjd}tjdjtjd}tjddgjtjd}tjdgjjd}tjddgjtjd}||||||gS)N)rr#r#r%r#rfrr#r$rr'rrZuint8rWZint32)rXrrinput_global_scalerK output_scaler8rrr*ws0    zIAllReduceFusedRMSNormStaticQuantNVFP4Pattern.register..get_inputsrXrrrKrrc sDt|}tt|||jd}tt||d||d}|d||dfS)NrrrxrXrZ input_scaler)r rrrSTATIC_FP4_QUANT_OP) rXrrrKrrrrrr8rrr9szFAllReduceFusedRMSNormStaticQuantNVFP4Pattern.register..patternrc sPt|}ttf||||||jtjj|d j }|d|d|dfS)Nrr$r) rrrrrrrkARResidualRMSNormFP4Quantrr)rXrrrKrrrrr8rrr?s$  zJAllReduceFusedRMSNormStaticQuantNVFP4Pattern.register..replacementrrrr8rrFus4  z5AllReduceFusedRMSNormStaticQuantNVFP4Pattern.registerrrrrrrg rcr) /AllReduceFusedAddRMSNormStaticQuantNVFP4Patternz This pattern replaces the allreduce + rms norm (with residual) + static nvfp4 quant with fused flashinfer implementation. Applies to o_proj + rmsnorm after attn + quant and mlp + rmsnorm + quant before attn. rrrrcrrrrrrrrrz8AllReduceFusedAddRMSNormStaticQuantNVFP4Pattern.__init__r+c sfdd}dtjdtjdtjdtjdtjdtjf fd d }dtjdtjdtjdtjdtjdtjf fd d }t|||tj|dS) Ncstjddgjjd}tjddgjjd}tjddgjjd}tjdjtjd}tjddgjtjd}tjddgjtjd}||||||gS)Nr#r%rrrr$r)rXrrKrrrr8rrr*s8    zLAllReduceFusedAddRMSNormStaticQuantNVFP4Pattern.register..get_inputsrrrXrrKrc sHt|}tt|||jd}tt||d||d}|d|d|dfS)Nrrrr)r rrrr) rrrXrrKrrrrr8rrr9s*zIAllReduceFusedAddRMSNormStaticQuantNVFP4Pattern.register..patternc sFttf||d|||jtjj|d j}|d|d|dfS)Nrr$rr)rrrrrrrr)rrrXrrKrrr8rrr?s" zMAllReduceFusedAddRMSNormStaticQuantNVFP4Pattern.register..replacementrrrr8rrFs,  z8AllReduceFusedAddRMSNormStaticQuantNVFP4Pattern.registerrrrrrrrrcs:eZdZdeffdd ZdejfddZddZZ S) AllReduceFusionPassrzcst|d|_t|_|jdkrdStdd|_|jdur!dS|j|_ t j |_ t }|jtjk}tdur@tddS|jtvrNtd|jdStt|jt|j |j|r^dnd|jjj}tj||j||j |j |d \|_}|at||j|||jjjd |_ d D]b}t!||j|j"|j #|jt$||j|j"|j #|jt%&d rt'||j|j"|j #|jt(||j|j"|j #|jt)||j|j"|j #|jt*||j|j"|j #|jtj+j,j-.qd |_dS)NTrZall_reduce_fusion_passr{zTFlashinfer is not installed or comm module not found, skipping allreduce fusion passz>Flashinfer allreduce fusion is not supported for world size %sr$r)Ztp_rankrrrgroupr)rr/rrr)gh㈵>gư>dF)/r|rdisabledr rrr}Z model_configZget_hidden_sizerrr=rr r~rrWrrwarningrrrrZcompilation_configZ pass_configZ!fi_allreduce_fusion_max_token_numZ1trtllm_create_ipc_workspace_for_all_reduce_fusion ipc_handlesrrZ enable_fusionrrrrFrr Zhas_device_capabilityrrrr _inductorpattern_matcherZ_seen_patternsclear)rrzrrZ max_num_tokenZworkspace_tensorrrrrr!s                  zAllReduceFusionPass.__init__rcCsN|jrdS|||d|j|}td|||d|dS)NZbefore_all_reduce_fusion_passzReplaced %s patternsZafter_all_reduce_fusion_pass)rrrr}rrrrrrrrrs     zAllReduceFusionPass.__call__cCs*|jrdStdurt|j|jdSdSr)rrZ+trtllm_destroy_ipc_workspace_for_all_reducerrr8rrr__del__szAllReduceFusionPass.__del__) rrrrrrrrrrrrrrrsb r)NNNN)Simportlib.utilrtypingrrZtorch._inductor.pattern_matcherrrrBZtorch.fxrZ*torch._higher_order_ops.auto_functionalizerrZ#torch.distributed._symmetric_memoryrZ vllm.configrZvllm.distributedrr Zvllm.distributed.parallel_stater r Z vllm.loggerr Zvllm.platformsr Z vllm.utilsrZvllm_inductor_passrZ fp8_dtyperTZflashinfer.commZcommrhasattr ImportErrorrrr1r5rr4Z ALLREDUCE_OPrtrrrrrrrrrr"rHrRrdrmrvryrMiBrrrArrrrrrrrrrrrrrrrrrrs$                   "&197? +      n       !AAOX\\