o )i!@sxddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z ddl mZe eZGd d d eZdS) N)Iterable)OptionalUnion)auto_functionalized) init_logger)is_func)VllmInductorPassc @seZdZdZdejjfddZdeejj e ejj ffddZ ddejjd ejj d e e eejj effd eeeejj efd ffd dZd ejj d e e eejj efffddZd ejj de e ejj ffddZ ddejjd ejj d eeeejj efd ffddZdS)FixFunctionalizationPassa" This pass defunctionalizes certain nodes to avoid redundant tensor copies. After this pass, DCE (dead-code elimination) should never be run, as de-functionalized nodes may appear as dead code. To add new nodes to defunctionalize, add to the if-elif chain in __call__. graphc CsD|||dg|_d}|jD]}t|tsq|j}|jd}|tj j j j krl|d}|jdjd}| |D]"\}} | jD]} t| tj jjj rY| ||| qD|| q=|||||n|tj j jj krddd} |||| nr|tj j jj krddd} |||| n]|tj j jj krdddd } |||| nG|tj j jj tj j jj fvrd di} |||| n-|tj j jj krd di} |j||| d d n|tj j jj krd di} |j||| d d nq|d 7}q||dt|j} |jD]}||qt d|| ||d|!dS)NZbefore_fix_functionalizationrqueryinputZresidual)rresultscale)rrr)rr args)rr rZ$before_fix_functionalization_cleanupz,De-functionalized %s nodes, removed %s nodesZafter_fix_functionalization)"beginZ dump_graphnodes_to_removenodesrrkwargsrtorchopsZ_CZrotary_embeddingdefault getitem_usersitemsusersZatenZ slice_scatterreplace_all_uses_with_removeinsert_defunctionalizedZfused_add_rms_normdefunctionalizeZ#fused_add_rms_norm_static_fp8_quantZ rms_norm_dynamic_per_token_quantZrms_normZrms_norm_static_fp8_quantZ silu_and_mulZsilu_and_mul_quantlenZ erase_nodeloggerdebugZ end_and_log) selfr countnoderZ at_targetr Zmm_nodeidxuserZuser_of_getitem mutated_argsZ count_removedr+r/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/compilation/fix_functionalization.py__call__s~                      z!FixFunctionalizationPass.__call__ node_or_nodescCs.t|tjjr|j|dS|j|dS)zM Stage a node (or nodes) for removal at the end of the pass. N) isinstancerfxNoderappendextend)r%r.r+r+r,rmsz FixFunctionalizationPass._removeNr'r*r.cCs*||||j|||d||dS)z De-functionalize a node by replacing it with a call to the original. It also replaces the getitem users with the mutated arguments. See replace_users_with_mutated_args and insert_defunctionalized. rN)replace_users_with_mutated_argsr r)r%r r'r*rr+r+r,r!ws z(FixFunctionalizationPass.defunctionalizecCsP||D]\}}||}t|tr|j|n|}||||qdS)a7 Replace all getitem users of the auto-functionalized node with the mutated arguments. :param node: The auto-functionalized node :param mutated_args: The mutated arguments, indexed by getitem index. If the value of an arg is a string, `node.kwargs[arg]` is used. N)rrr/strrrr)r%r'r*r(r)argr+r+r,r4s    z8FixFunctionalizationPass.replace_users_with_mutated_argsreturncCs2i}|jD]}t|tjr|jd}|||<q|S)z Returns the operator.getitem users of the auto-functionalized node, indexed by the index they are getting. r)rroperatorgetitemr)r%r'rr)r(r+r+r,rs   z&FixFunctionalizationPass.getitem_userscstts Jdd|5jd}|dur%|j|jdntfdd|D}|j||dWddSWddS1sJwYdS) a> Insert a new defunctionalized node into the graph before node. If one of the kwargs is 'out', provide args directly, as node.kwargs cannot be used. See https://github.com/pytorch/pytorch/blob/a00faf440888ffb724bad413f329a49e2b6388e7/torch/_inductor/lowering.py#L351 :param graph: Graph to insert the defunctionalized node into :param node: The auto-functionalized node to defunctionalize :param args: If we cannot use kwargs, specify args directly. If an arg is a string, `node.kwargs[arg]` is used. z%node must be auto-functionalized, is z insteadrN)rc3s(|]}t|trj|n|VqdSN)r/r5r).0r6r'r+r, s zCFixFunctionalizationPass.insert_defunctionalized..r)rrZinserting_beforerZ call_functionrtuple)r%r r'rfunctionr+r<r,r s     "z0FixFunctionalizationPass.insert_defunctionalizedr:)__name__ __module__ __qualname____doc__rr0ZGraphr-rr1rrdictintr5rr>r!r4rr r+r+r+r,r sH Q   "r )r8collections.abcrtypingrrrZ*torch._higher_order_ops.auto_functionalizerZ vllm.loggerrZfx_utilsrZvllm_inductor_passr r@r#r r+r+r+r,s