o p iD@sddlZddlZddlZddlmZddlZddlmZddl Z ddl m Z ddl m Z ddl mZddlmZddlmZdd lmZgd ZeeZeejejd d ZeZeee ed Z!dZ"dZ#e j$j%j&e j$j%j'gZ(dgZ)d8ddZ*ddZ+ddZ,d9ddZ-ddZ.d:dd Z/d!d"Z0d#d$Z1d%d&Z2d'd(Z3d)d*Z4d+d,Z5d-d.Z6d/d0Z7d1d2Z8 3 d;d4d5Z9d6d7Z:dS)<N) OrderedDict) text_format)base)core)Program) framework_pb2)draw_block_graphviz)io_utils) load_program save_programprogram_type_transcheck_saved_vars_try_dump parse_programcheck_pruned_program_varsgraphvizz)%(asctime)s - %(levelname)s - %(message)s)fmtzvars_persistable.logz vars_all.logzops.logZ lookup_tableFcCs|rt|St|SN)load_program_textload_program_binary)model_filenameis_textrn/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/paddle/incubate/distributed/fleet/utils.pyr ;sr cCs<t|d }|}Wdn1swYt|S)z$load program from binary string filerbN)openreadrparse_from_string)rfZprogram_desc_strrrrrAs   rcCsTt|d }|}Wdn1swYt}t||t|S)z*load program from human-readable text filerN) rrrZ ProgramDescrZMergerrZSerializeToString)rrZprogram_desc_textZ prog_descrrrrHs   r __model__cCs|r!t|d}|t|WddS1swYdSt|d}||jWddS1s:wYdS)Nwwb)rwritestrdescZserialize_to_string)programrrrrrrr Rs " "r c Csd}dd|D}t|}t|}td||D]^}||}|jtvr+|Sz ||}Wn t yT}zt d|dt |WYd}~qd}~ww|j |j ksa|j |j kr{t d|d|j d |j d |j d |j d }q|S) NTcSs g|] }t|r|j|fqSr)r is_persistablename.0vrrr ^s z-check_pruned_program_vars..z$persistable vars in pruned program: znot find variable 'z)' in train program. please check pruning.z variable: z% not match. in pruned program shape: z dtype:z, in train program shape: z dtype: F) list_varsrlistloggerinfotypefeed_fetch_type_list global_blockvar ValueErrorerrorshapedtype) Z train_progZ pruned_progZis_matchZ pruned_varsZpruned_vars_namevar_namer3Ztrain_prog_varerrrr[s<      (rdebugcCsbtj||d}tj||d}t||ddd|d|g}tj|tjtjtjd}|dS)Nz.dotz.pdf)pathdotz-Tpdfz-o)stdinstdoutstderr)osr<joinr subprocessPopenPIPEwait)block output_dirfilenameZdot_pathZpdf_pathcmdprrrrs  rcCsFttj|||}|r|dn|d}t|tj||d||S)Nz.binz.pbtxt)r rAr<rBr )Zprog_dirZprog_fnrprogZ prog_out_fnrrrr sr cCs |jdd|giid|iddS)NsaveX file_pathr0ZinputsZoutputsattrsZ append_oprGr3r<rrrappend_save_ops rUcCs |jdid|gid|iddS)NloadZOutrPrQrSrTrrrappend_load_ops  rWc Cst}t}t|}t|}t|#tjj|||d} t | | ||j ||igdWddS1s.) isinstancer-tuplerstripsplitlenappendnparrayZreshape) batch_sizefndimr`r6_tempxlinefieldstmprrrreaders(      r|cCs0g}t|D]\}}|t||||q|Sr) enumeraterqr|)rtfeeded_vars_dimsfeeded_vars_filelist batch_feedirurrrfeed_gensrc% st}t|} tj} t| l|rt|||}tjj j || ||d\} } dd|D} |D]B}t |j }|dusIJd|j t|j}|j | vs]J|j d| |j }||krvtd|d|j d|d q4d d | D}| std |std | }| }jdur| jkrtd| djdj}}g}t|jD]\}}|jd|jdkr||q|dddD]}||q|jdur.||jkr.td|d|jdfdd |jD}}g}t|jD]\}}|jd|jdkr||q|dddD]}||q%t dd|D}gt!jt!j"krOt!j#ksRJJt$t!jD]G}%j|}t&j"|t't(fswj"|f}nt(j"|}|j"|<|jdd}||krtdj|d|d|qYj)s#t*dt$t!jD]^}%j|}|j+dkr߈tjtj,,|gt'j"|Rj#|d q|j+dkr tjtj,,|gt'j"|Rj#|d }t-|dg|g|qtd!| j.fd"dt|D||d#}n7t*d$j)dfd%d t$t!jD} tj/| |d&}!t0|j"j)}"|"g}#| j.|!1|#||d#}t|D]\}}$t*d'|$j t*d(||q^|WdS1swYdS))N)rZparams_filenamecSsi|] }|jt|jqSr)r'rmr$r6)r)each_varrrr sz'try_load_model_vars..zcan't not find var: zMUST in var listzFShape not matching: the Program requires a parameter with a shape of (z(), while the loaded parameter (namely [ z ]) has a shape of (z).cSg|]}|jqSrr'r(rrrr+ z'try_load_model_vars..zno feed targets in program.zno fetch targets in program.z;feed vars in program and config are diff: feed in program: z. feed in config .FrYz=fetch vars in program and config are diff: fetch in program: z. fetch in config csg|] }|qSr)r2r3r)r)inference_programrrr+)s fetchcss|]}|jdkVqdS)rN) lod_levelr(rrr 8sz&try_load_model_vars..rLzfeed variable 'z)' shape not match. infer program shape: z. feed tensor shape: zgenerate random feed vars.r)r7zOvars with lod_level >= 2 is not supported now in this infer program check tool.csi|] \}}||qSrr)r)rr') feed_tensorsrrrws)rYrZ return_numpyzload feed vars from files: cs g|] }j|qSr)r2r3feeded_vars_namesr) feed_configrrrr+s )Z feed_listrdzfetch_targets name: zfetch_targets: )2rr[r\rZScopeZ scope_guardr r^r_ioZload_inference_modelZ global_scopeZfind_varr'rrrsZ get_tensorr6get RuntimeErrorr.warningrr2r}opsr$Z set_is_targetr0rqZ _remove_opZfetch_vars_namesallrpr~Zfeeded_vars_typesranger3rlr-rmrr/rrandomZcreate_lod_tensorraZ DataFeederrrY)%dump_dir dump_prog_fnis_text_dump_programrtr fetch_config save_filename saved_paramsrdrescopeZfeed_target_namesZ fetch_targetsZorig_para_shaperZvar_tempZ new_shapeZ orig_shapeZfetch_targets_namesrZZfeed_name_listr2Zneed_to_remove_op_indexropindexrr3Z tensor_shapeZ var_shapetresultsZ feed_varsZfeederrslotsr*r)rrrrtry_load_model_varssJ                                &rcCsNt}|jD]}|jtvr$|j|vr$td|jd||jqdS)Nzfind op type 'z@' in program, please check if your program is pruned correctly !)setr2rr0not_expected_op_typesr.radd)rMZ op_types_setrrrrcheck_not_expected_opss  rrLc Cs^ttj|||}dd|D}tddd|Dt|t||||||||S)NcSg|] }t|r|qSrr r&r(rrrr+  z-check_saved_vars_try_dump..z"persistable vars in dump program: cSrrrr(rrrr+r) r rAr<rBr,r.r/rr) rrrrrrtrZ dump_progrrrrr s( r c Csi}dd|D}dd|D|d<ttj|td }|d|dD]}|t||dq(Wdn1sAwYt|}dd|D|d <ttj|t d }|d |d D]}|t||dqiWdn1swY| j }d d|D|d <ttj|t d!}|d |d D]}|t||dqWddS1swYdS)NcSrrrr(rrrr+rz!parse_program..cSs:g|]}t|jt|jt|jt|jt|jdqS))r'r6rr7r0)r#r'r6intrr7r0r(rrrr+spersistable_varsr zpersistable vars:  cSsRg|]%}|jtvrt|jt|jt|jt|jdn t|jt|jdqS))r'r6rr7)r'r0)r0r1r#r'r6rrr7r(rrrr+s all_varsz all vars: cSs&g|]}|jt|jt|jdqS))r0input_arg_namesoutput_arg_names)r0r#rr)r)rrrrr+srzops: ) r,rrAr<rBpersistable_vars_out_fnr"r#r-all_vars_out_fnr2r ops_out_fn) r%rHoutputrrr3rrrrrrrsH              "r)F)rF)r:r;)rLN);loggingrArC collectionsrnumpyrrZgoogle.protobufrr^rZ paddle.baserZpaddle.base.frameworkrZpaddle.base.protorZ*paddle.distributed.fleet.base.util_factoryrZpaddle.frameworkr __all__ getLogger__name__r.setLevelINFO Formatter formatter StreamHandlerch setFormatter addHandlerrrrZVarDescZVarTypeZFEED_MINIBATCHZ FETCH_LISTr1rr rrr rrr rUrWrgrhr|rrrr rrrrrs^               $   D !