o * iSK@sddlZddlZddlZddlmZddlmZmZmZes"er-ddlm Z ddZ ndZ ddZ gdZ da Gd d d Z dd dZddZddZddZddZddZddZdS)N)core) CUDAPlaceis_compiled_with_cudais_compiled_with_rocm) CUDAGraphcCdS)NTrrre/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/paddle/device/cuda/graphs.pyis_cuda_graph_supportedr cCr)NFrrrrr r #r )global thread_localrelaxedc@s@eZdZdddZddZddZd d Zd d Zdd dZdS)rNr cCs`tdusJdd|_|durttjdd}t|}||_|tvs%Jt ||_ ||_ dS)NzFCUDA Graph is only supported on PaddlePaddle compiled with NVIDIA GPU.ZFLAGS_selected_gpusr) CoreCUDAGraph_graphintosenvirongetr_place ALL_MODESindex_mode_pool_id)selfZplacemodeZpool_idZ device_idrrr __init__,s    zCUDAGraph.__init__cCst|j|j|jdSN)rZbegin_capture_with_pool_idrrrrrrr capture_begin:s zCUDAGraph.capture_begincCst|_dSr)r end_capturerrrrr capture_end?zCUDAGraph.capture_endcC|jdSr)rreplayrrrr r$Br"zCUDAGraph.replaycCr#r)rresetrrrr r%Er"zCUDAGraph.resetcCs\t|ttfs |j}tj|ddtj|sJd|d|dur%d}|j ||dS)NT)nameexist_okz The dirname z should be a directoryi) isinstancestrbytesr&rmakedirspathisdirrprint_to_dot_files)rdirnameflagsrrr r.Hs zCUDAGraph.print_to_dot_files)Nr Nr) __name__ __module__ __qualname__rrr!r$r%r.rrrr r+s rr defaultc s|tvsJtsCddlm}tt}td7a|dkrd}n|dkr(t}nt d|||dt|d|fdd Sdd l m }dd l m }|}t|r\|j} n|} || _|dkrjd| _|S|dkrut| _|St||r|jj| _|S|j| _|S) Nr)_cuda_graph_guardr4newzMmemory_pool should be one of default or new under static graph mode, but got ;cs|i|Srr)argskwargsfunctionrr gsz!wrap_cuda_graph..) to_static)Layer)rpaddleZin_dynamic_modeZpaddle.base.frameworkr5r) cuda_graph_idrZgen_new_memory_pool_id ValueErrorZ paddle.jitr>Z paddle.nnr?r(forwardZ_cuda_graph_capture_modeZ_cuda_graph_pool_id) r<rZ memory_poolr5Zgraph_idmemory_pool_idr>r?Z new_functionZ mock_funcrr;r wrap_cuda_graphTsH          rEcCsX||j||j||j||j||j | |j | |j dS)z copy var desc from src to dst :param dst: framework.VarDesc(cpp), dst var desc, cpp VarDesc instance :param src: framework.VarDesc(cpp), src var desc, cpp VarDesc instance :return: no return N) set_shapeshapeZ set_dtypeZdtypeZ set_lod_levelZ lod_levelset_typetypeZset_persistable persistableZset_is_parameterZ is_parameterZset_stop_gradient stop_gradient)dstsrcrrr copy_var_descs      rNcCsDg}t|jD]\}}||krq|jD]}||qqtt|S)a1 find all inputs of ops after an idx, used to determine the logical output of a cuda graph section :param block: framework.Block, the original block :param begin_idx: int, from which idx (not include) to find the later ins :return: a list of inputs names for all ops behind begin_idx ) enumerateopsinput_arg_namesappendlistset)blockZ begin_idxinsidxopin_namerrr all_inputs_of_later_ops   rZcCsNtj}|}|}g}g}tj}t||d} |D]}} | jD]=} | | } |j | j d} t | | || dkrN|| dkrN|| q$| | dkra|| dkra|| q$| jD]"}| |} |j | j d} t | | ||dkr||qe|j }|| j ||| |q||||gfS)a 1. Construct a new program for corresponding section 2. Find all the logical inputs and outputs of a program section :param section: list, one cuda graph section, list of ops :param origin_program: framework.Program, origin program :param section_idx: list, the section ops' idx corresponding to the cuda graph section, a list of idx :return: a new program for the cuda graph section the logical ins and outs of the cuda graph section asciir)r@ZstaticZProgram global_blockrop_proto_and_checker_makerkOpRoleAttrNamerZrQvarZdescr&encoderNcountrRremoveZoutput_arg_namesZ append_opZ copy_fromZ _set_attrattrZ_sync_with_cpp)sectionorigin_program section_idxprogramrU origin_blockrVoutsop_role_attr_nameZ later_insrXrYr`Z new_var_descZout_nameZ new_op_descrrr #construct_program_and_find_ins_outss:              rlcCs|}g}g}d}g}g}g}g}d} tj} ttjjj} ttjjj} | | B} t|j D]\}}|j dks?|j dkrH|j dusHJd| drT| drTd}|j dur-t|j tsdJd |j d }t|d kstJd t|d }|| krt|dkrt|t|ksJd|D]H}t| | | kpt| | | k}|j dkp|j dk}|s|rg}g}t|t|ksJdt|dkr||||g}g}nqtt|D]}||||||qg}g}||||q0g}g}|} t|t|ksJdt|dkr&|||||g}|g}q0||||q0t|t|ksEJdt|dkrV|||||||fS)a get all sections that should run under cuda graph and the corresponding idx :param program: framework.Program, the original program :return: A list of cuda graph sections and the corresponding ops' idx in the block. The program is under is test or not. Fr[Zconditional_blockwhileNz9Cuda graph not support conditional block op and while op.is_testTzcuda_graph_attr should be a strr8zccuda graph attr should have three fields: cuda graph mode, cuda graph memory pool id, cuda graph idrz@len of internal section should be equal with len of internal idxz-num of section's op is not equal with the idxz0num of section's op is not equal with num of idx)r]rr^r_rZOpRoleZLossZBackwardrOrPrI_cuda_graph_attrZhas_attrrdr(r)splitlenrRrange)rhrUcuda_graph_sections sections_idxrnZinternal_sectionZ internal_idxZcurrent_sectionZ current_idxZcurrent_cuda_graph_idrkZ loss_op_roleZbackward_op_roleZloss_grad_op_rolerWrXZcuda_graph_attrsZlocal_cuda_graph_idZ internal_opZ loss_relatedZsub_block_relatedirrr get_cuda_graph_sectionss                     rxcCs2|d}|d}|d} |} t|D] } | j| ddqd} d} |D]}|jdur=|jd}|d} t|d} nq$| durF| dusJJd| jdt|tj j j d d d }| jd t|tj j j d d d }t j||}| j| d d |i|||d|dt|j||| | d||d ddS)a Use section_program and ins_and_outs to initialize a run_program_op, and replace the section_idx marks ops in the origin program. :param ins_and_outs: list, the logical ins and outs of the section program :param section_program: framework.Program, the partial program need to run under cuda graph :param section_idx: list, the idx need to be removed from origin program :param origin_program: framework.Program, the origin program :param cuda_graph_section: list, the ops in current sections, used to get the mode, memory pool id and is_test :param order: int, the order of current section, used to create unique cuda graph var :param is_test: bool, the program is running under is_test or not :return: no return rr6F)syncNr8z>mode and memory pool id should be specified in cuda graph attrZ cuda_graph_T)r&rIrJrKZprogram_out_scope_Z run_programX)ZOutZOutScoper) r]Zstart_op_indexZ end_op_indexrn program_idZcuda_graph_capture_modeZcuda_graph_pool_idZuse_interpretorcoreZforward_global_blockZbackward_global_block)rIZinputsZoutputsattrs)r]reversedZ _remove_oprqrrrZ create_varr)rZVarDescZVarTypeZRAWZ STEP_SCOPESr@utilsZ _hash_with_idZ _insert_oprsrP) ins_and_outssection_programrgrfZcuda_graph_sectionorderrnrVrjZ insert_idxrirWrrDrXr|Zcuda_graph_varZ out_scope_varr{rrr replace_cuda_graph_sectionRsf        rc Cst|jdkr tdt|\}}}t|t|ksJdg}g}tt|D]}t|||||\}}||||q)t|t|ksNJdttt|D]}t |||||||||||dqV|S)z replace the ops marked with cuda_graph_attr to run_program_op to use cuda graph :param program: framework.Program, the program to be transformed :return: the cuda graph section program, user should hold these programs! r6zSub block(s) has been detected in the program. Cuda graph not support op with sub block, and it will only handle the global block.z@num of cuda graph sections is not equal with num of idx sectionszJthe num of cuda graph sections should be equal with the num of new program)rrn) rsblockswarningswarnrxrtrlrRr}r) rhrurvrnrZsection_programsrwrZins_outsrrr cuda_graph_transforms@    r)r r4)rrr@Z paddle.baserZpaddle.base.corerrrrrr rrArErNrZrlrxrrrrrr s(     ),4| Y