o p i9@@sddlZddlZddlZzddlmZWn ddlmZYddlZddlmZm Z ddl m Z m Z ddl mZmZmZmZddlmZd d lmZdd lmZdd lmZmZmZeeejd dZgdZddgZgdZgdZ dZ!ddZ"ddZ#GdddZ$GdddZ%GdddZ&Gdd d Z'Gd!d"d"Z(Gd#d$d$Z)Gd%d&d&Z*Gd'd(d(Z+Gd)d*d*e$Z,Gd+d,d,Z-Gd-d.d.Z.Gd/d0d0Z/Gd1d2d2Z0Gd3d4d4Z1dS)5N)tqdm)IrGraphIrNode)_get_paddle_placecore)Programdata program_guard scope_guard) unique_name) get_logger)utils) SUPPORT_ACT_QUANTIZATION_OP_DICTSUPPORT_QUANTIZATION_OP_DICT#SUPPORT_WEIGHT_QUANTIZATION_OP_DICTz&%(asctime)s-%(levelname)s: %(message)s)fmt)fake_quantize_abs_maxfake_quantize_range_abs_max$fake_quantize_moving_average_abs_max"fake_channel_wise_quantize_abs_maxfake_dequantize_max_abs$fake_channel_wise_dequantize_max_abs)/fake_quantize_dequantize_moving_average_abs_maxZ-fake_channel_wise_quantize_dequantize_abs_maxZ fake_quantize_dequantize_abs_max)conv2ddepthwise_conv2dconv2d_transposegMbP?cCsVt|tjs Jd|dusJd|dusJd||}|||dS)Nz(The type of value should be numpy array.The scope cannot be set None.The place cannot be set None.) isinstancenpZndarrayvarname get_tensorset)var_nodevaluescopeplacetensorr,s/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/paddle/static/quantization/quantization_pass.py_init_var_nodeBsr.cCs4d}t|D]}||j|}|o| }q|S)zI Analyse the real inputs of the op node are all not persistable. T)r_get_op_input_var_names_find_node_by_nameinputs persistable)graphop_nodeZis_input_all_not_persistablevar_namein_noder,r,r-_is_input_all_not_persistableLsr7c@seZdZdZdddddddddggddddddddfd d Zd d Zd dZddZddZddZ ddZ ddZ ddZ ddZ ddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*ZdS)+QuantizationTransformPasszk Quantize the ops that have weights. Add quant and dequant ops for the quantized ops's inputs. Nabs_max'? skip_quantrrmulcCs||_t||_||_||_| |_| |_| |_| |_||_ ||_ ||_ gd}|dks/Jd||vr;t d|d||vrGt d|d||_ ||_||_||_| |_|jD]}|ttvsiJ|dqYd d |jD|_||_d |_i|_i|_d S) a Constructor. Args: scope(static.Scope): When activation use 'range_abs_max' as the quantize type, this pass will create some new parameters. The scope is used to initialize these new parameters. place(static.CPUPlace|static.CUDAPlace|str): place is used to initialize new parameters described above. If it's string, It can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. weight_bits(int): quantization bit number for weights, the bias is not quantized. activation_bits(int): quantization bit number for activation. activation_quantize_type(str): quantization type for activation, now support 'abs_max', 'range_abs_max' and 'moving_average_abs_max'. If use 'abs_max' mode, the quantization scale will be calculated dynamically each step in both training and testing period. If use 'range_abs_max', a static quantization scale will be calculated during training and used in inference. weight_quantize_type(str): quantization type for weights, support 'abs_max' and 'channel_wise_abs_max'. The 'range_abs_max' usually is not used for weight, since weights are fixed once the model is well trained. window_size(int): the window size for 'range_abs_max' quantization. moving_rate(float): the param for 'moving_average_abs_max' quantization. skip_pattern(str or str list): The user-defined quantization skip pattern, which will be presented in the name scope of an op. When the skip pattern is detected in an op's name scope, the corresponding op will not be quantized. quantizable_op_type(list[str]): List the type of ops that will be quantized. Default is ["conv2d", "depthwise_conv2d", "mul"]. The quantizable_op_type in QuantizationFreezePass and ConvertToInt8Pass must be the same as this. weight_quantize_func(function): Function that defines how to quantize weight. Using this can quickly test if user's quantization method works or not. In this function, user should both define quantization function and dequantization function, that is, the function's input is non-quantized weight and function returns dequantized weight. If None, will use quantization op defined by 'weight_quantize_type'. Default is None. act_quantize_func(function): Function that defines how to quantize activation. Using this can quickly test if user's quantization method works or not. In this function, user should both define quantization and dequantization process, that is, the function's input is non-quantized activation and function returns dequantized activation. If None, will use quantization op defined by 'activation_quantize_type'. Default is None. weight_preprocess_func(function): Function that defines how to preprocess weight before quantization. Using this can quickly test if user's preprocess method works or not. The function's input is non-quantized weight and function returns processed weight to be quantized. If None, the weight will be quantized directly. Default is None. act_preprocess_func(function): Function that defines how to preprocess activation before quantization. Using this can quickly test if user's preprocess method works or not. The function's input is non-quantized activation and function returns processed activation to be quantized. If None, the activation will be quantized directly. Default is None. optimizer_func(function): Function return a optimizer. When 'is_test' is False and user want to use self-defined quantization function and preprocess function, this function must be set. Default is None. executor(base.Executor): If user want to use self-defined quantization function and preprocess function, executor must be set for initialization. Default is None. Examples: .. code-block:: python >>> # The original graph will be rewrite. >>> import paddle.static as static >>> from paddle.static.quantization import QuantizationTransformPass >>> from paddle.base.framework import IrGraph >>> from paddle.framework import core >>> graph = IrGraph(core.Graph(static.Program().desc), for_test=False) >>> place = paddle.CPUPlace() >>> transform_pass = QuantizationTransformPass(static.global_scope(), place) >>> transform_pass.apply(graph) r:channel_wise_abs_max range_abs_maxmoving_average_abs_maxrAIThe activation quantization type does not support 'channel_wise_abs_max'.$Unknown activation_quantize_type : 'K'. It can only be 'abs_max' or 'range_abs_max' or 'moving_average_abs_max'.Unknown weight_quantize_type: 'e'. It can only be 'abs_max' or 'channel_wise_abs_max' or 'range_abs_max' or 'moving_average_abs_max'.# is not supported for quantization.cSg|]}|dqSZ_gradr,.0opr,r,r-  z6QuantizationTransformPass.__init__..N)_scoper_place _weight_bits_activation_bits _skip_pattern_weight_quantize_func_act_quantize_func_weight_preprocess_func_act_preprocess_func _optimizer_exe ValueError_activation_quantize_type_weight_quantize_type _window_size _moving_rate_quantizable_opslistrkeys_quantizable_grad_ops_is_test _global_stepcreate_var_map create_op_mapselfr)r* weight_bitsactivation_bitsZactivation_quantize_typeweight_quantize_type window_size moving_rate skip_patternquantizable_op_typeZweight_quantize_funcZact_quantize_funcZweight_preprocess_funcZact_preprocess_funcZoptimizer_funcexecutoris_test quant_typerNr,r,r-__init___sN_      z"QuantizationTransformPass.__init__c slt|ts Jdjdur|_tdd|Dgfdd}fdd}fd d }fd d }jsE||}|D]}| j vs[| j vr_||qKi|_ t t|d dd'}|D]}| j vr||s||r||||qoWdn1swY|D]}| j vr||r|||q||S)4 Quantize the graph for training process. According to weight and activation quantization type, the graph will be added some fake quantize operators and fake dequantize operators. Args: graph(IrGraph): the applied graph. Returns: None &graph must be the instance of IrGraph.NcSg|]}|qSr,r$rMpr,r,r-rOz3QuantizationTransformPass.apply..csd}tjtrdotfddjD}ntjtr6do5djdk}|rJ dd dddSdS) NF op_namescopec3"|] }|dvVqdSr}NrNattrrMpatternr4r,r-   zMQuantizationTransformPass.apply.._quant_preprocess..r=Twith_quant_attr r!rUrbrNhas_attranystrrfind _set_attr)r4 user_skippedrjrr-_quant_preprocesss"  z:QuantizationTransformPass.apply.._quant_preprocessc s|dd|dd|d}|j}|D]}||vr'q|vr4|}n|}|vr=q|vrEdnd}|rXjdurX|j||}n|shjdurh|j||}|r~j dur~|j ||} |q|sj dur|j ||} |q|vrj nj } |rjnj} | dkr|} | dko|d } | r| d n| } | tjvrd nd } |||| | |\}}|||g| g| |}n|||| | |\}}|||| |}||<||||qdS) Nquantization_typeqat_with_weightrTop_roleFrA matmul_v2trans_y_trans_yrr)rNrrr1r$input_arg_namesrX _insert_funcrYrVappendrWrSrTr^r]r_channelwise_quant_axis1_ops_insert_channel_quant_op_insert_channel_dequant_op_insert_quant_op_insert_dequant_opupdate_input_link)r3rNrr1r'dequant_var_noder$ is_weighttarget_out_node quant_bitsrtop_typer quant_axisquant_var_nodescale_var_nodedequantized_varspersistable_varsprocessed_varsrjr,r-_transform_forwards             z;QuantizationTransformPass.apply.._transform_forwardcsH|jD]}||vrq|vr!|}||||qdSN)r1r$rr)r3rNr'r)rr,r-_transform_backwards   z._transform_backwardcs>d}|jD]}||vrq|}|vrd}q|SNFT)r1r$r)rN has_weightr'r$)rr,r- _has_weights  z4QuantizationTransformPass.apply.._has_weight7Adding quant op with weight:|{bar}| {n_fmt}/{total_fmt}PtotalZ bar_formatZncols)r!rrers collections OrderedDictall_persistable_nodes_create_global_step all_op_nodesr$rardout_node_mapping_tablerlen_is_skip_quantupdateresolve_hazard) rjr3rrrropsrNtr,rr-applysP     q      zQuantizationTransformPass.applycCs|jdks |jdkrkd}|D] }||kr||_q|jdurm|j|tjjj dgtjjj d}t |t j dgdd|j|j||}|jddtjjjd d |id |id }||||||||_dSdSdS) NrBz@STEP_COUNTER@rr$var_typeshape var_dtypeZint64dtype incrementg?)steprXOutrattrsr1outputs)r^r] all_var_nodesr$rfcreate_persistable_noderVarDescVarType DENSE_TENSORZINT64r.r"zerosrQrRcreate_var_node_from_descr#create_op_nodeop_proto_and_checker_makerOpRoleForwardlink_to)rjr3Z counter_namenodeZglobal_step_inZglobal_step_outZ increment_opr,r,r-rsH       z-QuantizationTransformPass._create_global_stepcCsR|dkr ||||||S|dkr||||||S|dkr'||||||SdS)z7 Insert fake_quantize_op in the graph. r:rBrCN)_insert_quant_abs_max_op_insert_quant_range_abs_max_op'_insert_quant_moving_average_abs_max_op)rjr3r'r$rrtrr,r,r-rs   z*QuantizationTransformPass._insert_quant_opc Cs$|s J|d|j|||||d}||}|tj kr.d}n |tj kr8d}nd}z t |j |} Wn t jdg|d} Y|j||dg|d} t| | |j |j|jd||d d |i|| d d } ||| || ||| | || fS) z? Insert fake_quantize_abs_max op in the graph. is not a varrfloat64float32float16rrr) bit_lengthrrrOutScaler)is_varr$create_var_node_quantized_var_nametyperr_quantized_scale_namepaddlerrr"arrayrQfind_varr%rrr.rRrr) rjr3r'r$rrr scale_name data_type scale_valuer quant_op_noder,r,r-rsH    z2QuantizationTransformPass._insert_quant_abs_max_opcCs|s J|d|j|||||d}||}|tj kr.d}n |tj kr8d}nd}z t |j |} Wn t j tg|d} Y|j|tjjjdg|d} t| | |j |j|| } || d} || d } |js|jtd tjjj|jg|d}|tj krd}n |tj krd}nd}t|t j|jg|d|j |j|j| d <|| d <|j||j|d }|j d|| | d}|!|||!| ||!|||!|| |js|!|j||!|||| fS)zB Insert fake_quantize_range_abs_max on the graph. rrrrrrrrZInScalerscalesZIterZ OutScales)rnrrsrrr)"rr$rrrrrrrrrr"rrQrr%_SCALE_DEFAULT_VALUErrrrrr.rRrr#rer generater_rrfrr)rjr3r'r$rrrrrr scale_in_nodescale_out_noder1rZ scales_noderrr,r,r-r%s         z8QuantizationTransformPass._insert_quant_range_abs_max_opcCsf|j|||||d}||}|tjkr!d}n |tjkr+d}nd}z t |j | } Wn t j tg|d} Y|j|tjjjdg|d} t| | |j |j|| } || d} || d} |js|jtd tjjj|dgd }|tjkrd}n |tjkrd}nd}t|t jdg|d|j |j|jtd tjjj|dgd }t|t jdg|d|j |j||}||}|| d <|| d <|| d<|| d<||j|j|d}|jd|| | d}||||| ||||||| |js/|||||||||||||| fS)z+Insert fake_quantize_moving_average_abs_maxrrrrrrrrstater$rrraccumInStateInAccumOutStateOutAccumrrorsrrr)rrrrrrrrrr"rrQrr%rrrrrrr.rRrr#rer ronesr`rr)rjr3r'r$rrrrrrrrinsouts state_in_node accum_in_nodestate_out_nodeaccum_out_noderrr,r,r-r|s           zAQuantizationTransformPass._insert_quant_moving_average_abs_max_opc Cs@|s J|d|j|||||d}||}|tj kr.d} n |tj kr8d} nd} z t |j |} Wnt j||g| d} Y|j|||||g|d} t| | |j |j|jd|||j|dd |i|| d d } ||| || ||| | || fS) zL Insert fake_channel_wise_quantize_abs_max op in the graph. rrrrrrr)rrrsrrrr)rr$rrrrrrrrrr"rrQrr%rrr.rRrrer) rjr3r'r$rrrrrrrrrr,r,r-rsT    z2QuantizationTransformPass._insert_channel_quant_opc Cs|s J|d|j|||||d}d|d>d}|jdt||d||dd|id}| ||| ||| |||S) z9 Insert fake_dequantize_op in the graph. rrrr max_rangerrScalerr) rr$r_dequantized_var_namerrrrfloatr) rjr3r'rrrrr dequant_op_noder,r,r-rs$     z,QuantizationTransformPass._insert_dequant_opc Cs|s J|d|j|||||d}|jd|||d||dd|id}||||D]} || |q;||||S)zK Insert fake_channel_wise_dequantize_max_abs in the graph. rrr)rrrrZScalesrr) rr$rr rrrrr) rjr3r'Zscale_var_nodesrrrrrZscale_nr,r,r-r2s*   z4QuantizationTransformPass._insert_channel_dequant_opcCsd}|jD]}||}q||}|jD]}||}q||jvr.|j|}|S|r=|}||j|<|S||j }||j|<|S)z create a node that same with in_node in graph Args: graph(IrGraph): create node in graph. in_node(IrVarNode): create node that same with in_node. Returns: created new node ) r1r$rrgrcZ is_ctrl_varZcreate_control_dep_varrrr#)rjr3r6keyinpnew_noder,r,r-_create_new_nodePs       z*QuantizationTransformPass._create_new_nodec Csd}|jD]}||}q||}|jD]}||}qd}||jvr1|j|}d}n ||j}||j|<|rBdS|jD]}|||} | | |qE|jD]}|||} | || qW|jD]} | jD] } | ||| qnqidS)aZ copy op_node in source_graph to graph. And will run recursively for next ops that link to op_node's outputs. Args: graph(IrGraph): target graph to copy. source_graph(IrGraph): source graph to copy. op_node(IrOpNode): op node in source_graph. Returns: None rFTN) r1r$rrhrccreate_op_node_from_descrrNrr _copy_graph) rjr3Z source_graphr4rrZ has_createdZ new_op_noder6rr' next_op_noder,r,r-rjs2           z%QuantizationTransformPass._copy_graphc Cst}t}t||U||d<t|d|dd}||}||j|j<t|} |j sJ|j s>Jdd|_ | } | | Wdn1sTwYWdn1scwYt |j|j|Wdn1s~wYtt|j|j d} | | |j}| | |j}g} g} | D]}|jgkr|r| |q| D] }|jgkr| |q|jD] }||| |q| D]}|jD] }||| |qq| D] }||| |q|||}|||}||| j}|j}|D] }||||q|||||j s|jd }|||d }|jd }|||d }|||d }|j}||||d}|jD]}|d |vry|}qj|dur||||n||||D]}|||||r||||q|jd }|jd }|jd }| || || || || |jd | || ||S) a Insert a tmp program that returned by func between var_node and op. Args: graph(IrGraph): target graph to insert tmp program. func(Function): function to define a tmp program var_node(IrVarNode): node in target graph. op(IrOpNode): op in target graph. Returns: op's new input that replaces var_node _Z _tmp_inputr)rrz3optimizer_func must be set when graph is test graphFN)Zfor_testr@GRAD)!r r Zswitch_name_generator_guardr$r rrrmeanZ _for_testrZZ stop_gradientZminimizer rQr[runrrZGraphZdescr0rr1r2rrrrrZupdate_output_linkrsafe_remove_nodes)rjr3funcr'rNZ tmp_programZstartup_programr6out_nodeZlossZ optimizerZ tmp_graphZin_node_paramsZ in_op_noderr4Ztarget_in_noderZ loss_noderZop_outZ op_out_gradZop_gradZtarget_out_grad_nodeZ in_node_gradZin_node_grad_opZ op_grad_outZ mean_gradZ mean_out_gradZfill_constant_noder,r,r-rs                              z&QuantizationTransformPass._insert_funccC |dSzJ Return quantized variable name for the input `var_name`. .quantizedr,rjr5r,r,r-r  z-QuantizationTransformPass._quantized_var_namecCr zL Return dequantized variable name for the input `var_name`. .dequantizedr,r#r,r,r-r r$z/QuantizationTransformPass._dequantized_var_namecCr zW Return the scale name of quantized variable for the input `var_name`. @scaler,r#r,r,r-rr$z/QuantizationTransformPass._quantized_scale_namecCsfd}|dr|drd}|dvrt||rd}|dr1|ddkr1d}|S)zA Analyse whether the op node skips quantization. Fr=T)r?matmulrqat_without_weight)rNrrr$r7)rjr3r4is_skipr,r,r-rs  z(QuantizationTransformPass._is_skip_quant)__name__ __module__ __qualname____doc__rurrrrrrrrrrrrrr rrr,r,r,r-r8YsL M'.Wh4&} r8c@sreZdZ      dddZdd Zd d Zd d ZddZddZddZ ddZ ddZ ddZ ddZ dS)QuantizationFreezePassFr9roundr:Nc Cs|dusJd|dusJd||_||_t||_||_||_||_||_t|_ t |_ t |_t |_t |_t|_dS)a The freeze pass is used to adjust the quantize operator order, for example: 1) `activation -> quant -> dequant -> conv2d` will be frozen into `activation -> quant -> conv2d -> dequant` 2) `weight -> quant -> dequant -> conv2d` will be frozen into `weight -> conv2d`, and weight will be scaled offline. Args: scope(static.Scope): scope is used to get the weight tensor values. place(static.CPUPlace|static.CUDAPlace|str): place is used to restore the weight tensors. If it's string, It can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. bias_correction(bool): whether use bias correction for post-training quantization. https://arxiv.org/abs/1810.05723. weight_bits(int): quantization bit number for weights. activation_bits(int): quantization bit number for activation. round_type(str, optional): The method of converting the quantized weights value float->int. Currently supports ['round', 'adaround'] methods. Default is `round`, which is rounding nearest to the integer. 'adaround' is refer to https://arxiv.org/abs/2004.10568. weight_quantize_type(str): quantization type for weights, support 'abs_max' and 'channel_wise_abs_max'. The 'range_abs_max' usually is not used for weight, since weights are fixed once the model is well trained. quantizable_op_type(list[str]): This input param will be removed latter. The pass will process all quantized op, so it is not necessary to set the input param. Nrr )rQ_bias_correctionrrRrSrT _round_typer^_fake_quant_op_list_fake_quant_op_names_fake_dequant_op_list_fake_dequant_op_namesrr_op_input_rename_map_op_output_rename_map_quant_var_scale_mapr&_quantized_ops) rjr)r*bias_correctionrkrlZ round_typermrqr,r,r-ru8s$     zQuantizationFreezePass.__init__cCsdd|D}|}|D]}|}||jvr|dd}t|dr2||jvr2|j|}||vrH||j | dd}||j |<q| | dd}|j dvs[Jd|j d krd|d}|jd krs|jd krs|d}n|}||j |<|jd kr| |}d} |d r|d } ||jvr|j|t||| |j} t| } |jdurtj|| || |jd} t| } ||| | ||q|}|D]}|}||j!vr| ||q|}|D]4}|} | dr%| ddkr%|jdkr|tj"vrd nd} |#||| q|$||q|D] }|j%D]} | j&|j'vrE| } |j'| j&}|(| ||q-q(|)||*|S)z Adjust quantize/dequantize operators order for the inference process. Args: graph(IrGraph): the applied graph. Returns: None cSrxr,ryrzr,r,r-rOvr|z0QuantizationFreezePass.apply..rrrrrr#the dim of scale_v should be 1 or 2rrr:r1rTrkrrrA)+rrr$r5inputhasattrrrcr0routputr: _load_varndimsizer^tolistr3rNrrr;addr quant_tensorcopyrSr"r1r2bias_correction_w _restore_var!_remove_fake_quant_and_dequant_opr7r_insert_post_channel_dequant_op_insert_post_dequant_opr1rr9r_remove_unused_var_nodesr)rjr3rrr4Zop_nameZinput_arg_namescale_vparam_vrquantized_param_vZ op_node_descr'old_innew_inr,r,r-rls                          zQuantizationFreezePass.applycCsl||j|dd}||j|dd}|j|jvr%||j|j<n |j|j|j|j<||dS)Nrrr)r0rrBr1r@rr8r)rjr3r4kvr,r,r-rLs  z8QuantizationFreezePass._remove_fake_quant_and_dequant_opcCsJdd|D}|jD]S}|}||vrq |j|jvr2|}|j|j}|||||||} |j | } | |vrSt | t sMJd| dt | } q t | tsZJ|j | } q t|dkrrtd|d||j|d} |jtd tjjj| jdg| d }| tjkrd }n | tjkrd }nd }t || !||j"|j#|j$|%| | &| | d }d}|dvrt|jdd}|'(dr|')d}|j*d|j+|j,g|tj-j.j/|d| || gdd|id}|0| ||0| ||0|||0||||j1| j<|S)NcSrxr,ryrzr,r,r-rOr|zJQuantizationFreezePass._insert_post_channel_dequant_op..The scale of parameter z is not a list.r Only support one output, but op  has more than one output.r channel_scalerrrr)r)rr?x_num_col_dimsr)rrrr[rrr)2rr1r$rrr8 clear_outputsr_original_var_namer:r!rbr"rrroutput_arg_namesr\r0rrr rrrrrrrrrrr.astyperQrRrr rrNrrrrSrTrrrrr9)rjr3r4rrr'r$rSrToriginal_var_namerPrZroutput_var_nodeweight_scale_noderrr[rr,r,r-rMs                   z6QuantizationFreezePass._insert_post_channel_dequant_opcCsdd|D}d}d|jd>d}d|jd>d}|jD]`}|}||vr-q |j|jvrF|} |j|j} | | | | || |} |j | } | |vrp| | saJd| d| dkrgdn| } ||| 9}q ||9}t | ts{J|j | } q t|dkrtd|d ||j|d }|j|||||d }|jd t|tjjjd || dd|id}||||| ||||||j |j<|S)NcSrxr,ryrzr,r,r-rO:r|zBQuantizationFreezePass._insert_post_dequant_op..rrWz is not a float.gg:0yE>rXrYrrrr r rr)!rrSrTr1r$rrr8r\rr]r: _is_floatr!rrr^r\r0rrr rrrrrrrrrrr9)rjr3r4rr Z param_rangeZ act_ranger'r$rSrTr`rPrrarrr,r,r-rN9sd             z.QuantizationFreezePass._insert_post_dequant_opcCt|j|Srr"rrQrr%rjr$r,r,r-rCrz QuantizationFreezePass._load_varcC"|j|}|||jdSrrQrr%r&rRrjr$rr+r,r,r-rKuz#QuantizationFreezePass._restore_varczt|}|D]}|jD]}|q|jD]}|qq ddDttfdd|}||dS)NcSh|]}|jqSr,rrMnr,r,r- zBQuantizationFreezePass._remove_unused_var_nodes..c |jvSrrnrnZ all_used_varsr,r- zAQuantizationFreezePass._remove_unused_var_nodes..r&rr1rGrfilterrrrjr3rr4 input_node output_nodeZall_unused_varsr,rtr-rOy      z/QuantizationFreezePass._remove_unused_var_nodescCst|dr|dtd S|dr|dtd S|dr*|dtd S|dr8|dtd S|S)z4 Return the original variable name. z.quantized.dequantizedNr"r&r()endswithrr#r,r,r-r]s    z)QuantizationFreezePass._original_var_namecCr r%r,r#r,r,r-r r$z,QuantizationFreezePass._dequantized_var_namecCst|ttjtjtjfSr)r!rr"rrr)rjrVr,r,r-rcsz QuantizationFreezePass._is_float)Fr9r9r1r:N)r,r-r.rurrLrMrNrCrKrOr]r rcr,r,r,r-r07s$ 4p R9 r0c@s>eZdZdddZddZddZdd Zd d Zd d ZdS)ConvertToInt8PassNcCs4|dusJd|dusJd||_t||_dS)a6 Convert the weights into int8_t type. Args: scope(static.Scope): scope is used to get the weight tensor values. place(static.CPUPlace|static.CUDAPlace|str): place is used to restore the 8bits weight tensors. If it's string, It can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. quantizable_op_type(list[str]): This input param will be removed latter. The pass will process all quantized op, so it is not necessary to set the input param. Nrr )rQrrR)rjr)r*rqr,r,r-rus zConvertToInt8Pass.__init__c Csdd|D}|}i}|D]7}|drH|ddkrH|jD]!}|}||vrG||vr>|||}|||<|||||q&q| || |S)z Convert weights' type of the graph. After that, the data type of the graph weights is int8_t. Args: graph(IrGraph): the applied graph. Returns: None cSrxr,ryrzr,r,r-rOr|z+ConvertToInt8Pass.apply..rr) rrrNrrr1r$_convert_to_int8rrOr) rjr3rrZ input_mapr4r'r$ int8_var_noder,r,r-rs,     zConvertToInt8Pass.applycCsZ|d}|j|||tjjjd}||}|j || ||t j |S)Nz.int8r)r$rrrrrrZINT8rCrQr# _store_varr"int8)rjr3r'Zint8_var_node_namerrr,r,r-rs  z"ConvertToInt8Pass._convert_to_int8cCrdrrerfr,r,r-rCrgzConvertToInt8Pass._load_varcCs(|j|}||||jdSr)rQrr%r&r_rR)rjr$rrr+r,r,r-rszConvertToInt8Pass._store_varcrl)NcSrmr,rnror,r,r-rqrrz=ConvertToInt8Pass._remove_unused_var_nodes..crsrrnrnrtr,r-rurvz.rwryr,rtr-rOr|z*ConvertToInt8Pass._remove_unused_var_nodesr) r,r-r.rurrrCrrOr,r,r,r-r~s #  r~c@seZdZddZddZdS)TransformForMobilePasscCst|_t|_dS)z\ This pass is used to convert the frozen graph for paddle-mobile execution. N)r4r5r6r7rr,r,r-rus zTransformForMobilePass.__init__c Cs|}|D]b}|}||jvr:|d||}|jD]}|||q |jD]}|||q,| |||j vrh|d||}|jD]}|||qN|jD]}|||qZ| |q| |S)a+ Because paddle-mobile use `quantize` an `dequantize` as the names of quantize operator and dequantize operator, the `apply` function just realize this logic. Args: graph(IrGraph): the graph will be transformed. Returns: None quantizeZ dequantize) rr$r5set_typerrNr1rrrr7r) rjr3rr4r$Z quant_noderzr{Z dequant_noder,r,r-rs,           zTransformForMobilePass.applyN)r,r-r.rurr,r,r,r-rs rc@s0eZdZ     d ddZddZddZdS) OutScaleForTrainingPassNr<cCs4||_t||_||_||_tt|_||_ dS)aS This pass is used for calculating output scales of some operators. These output scales may be used by tensorRT or some other inference engines. Args: scope(static.Scope): The scope is used to initialize these new parameters. place(static.CPUPlace|static.CUDAPlace|str): The place is used to initialize new parameters. If it's string, It can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. moving_rate(float): The decay coefficient of moving average. The default value is 0.9. N) rQrrRr`rerbrrc _teller_set _scale_dict)rjr)r*rors scale_dictr,r,r-ru+s   z OutScaleForTrainingPass.__init__c CsTt|ts Jd|jdur||_g}|D]}||jvr'||qtt |dddm}|D]`}t |D]S}| |j |}|tjjjtjjjtjjjfvs_d|vr`q=|tjkrjd}n |tjkrtd}nd }z| |||Wq=|j||tjjjd g|d }|jdurz t|j|g} Wntjd g|d } Yntjd g|d } Yt || |j!|j"d |i} d|i} |js;|jt#$dtjjj|d gd} t | tjd g|d |j!|j"|jt#$dtjjj|d gd} t | tjd g|d |j!|j"|%| &}|%| &}| | d<| | d<|| d<|| d<|j'|j|()dd}|j*d|| | d}d}t |j dkr`|j d}|+|||+|||ru|+|||js|+| ||+| ||+|||+||q=|,q5Wd|S1swY|S)z Insert the `moving_average_abs_max_scale` op in order to calculate output scales of operators in the teller_set. Args: graph(IrGraph): the target graph. rwNz.Adding OutScale op:|{bar}| {n_fmt}/{total_fmt}rrrrrrrrrrrz scale_state@rz scale_accum@rrrrr)rorsrmoving_average_abs_max_scalerr)-r!rrersrr$rrrrr_get_op_output_var_namesr0rrrrrFP64FP32FP16rrrr _scale_namerrrr"rrr.rQrRr rrr#r`rNrrrr)rjr3Z target_opsrNrZoutput_var_namer6r scale_noderrrrrrrrZ scale_op_noderr,r,r-rEs                      xxzOutScaleForTrainingPass.applycCr E Return the scale name for the var named `var_name`. r(r,r#r,r,r-rr$z#OutScaleForTrainingPass._scale_name)NNr<NNr,r-r.rurrr,r,r,r-r*s  rc@s&eZdZdddZddZddZdS) OutScaleForInferencePassNcCs||_tt|_dS)a  This pass is used for setting output scales of some operators. These output scales may be used by tensorRT or some other inference engines. Args: scope(static.Scope): The scope is used to initialize these new parameters. N)rQrbrrcr)rjr)r,r,r-rusz!OutScaleForInferencePass.__init__c Cs@t|ts Jd|}|D]}||jvrt|}|D]y}||j|}|j dusA| t j jjt j jjt j jjfvrBq||}|j|}|dusYJd|dt|d} |dt| t||} | dus|J|d|| dt| dd t| |d d qq||S) z Get output scales from the scope and set these scales in op_descs of operators in the teller_set. Args: graph(IrGraph): the target graph. rwNz Can not find z variable in the scoperZ out_thresholdz is not the output of the oprZ _thresholdrT)r!rrr$rrrr0rrr#rrrrrrrrrQrr"rr%rNrrZ_get_output_name_indexrr) rjr3Zop_nodesr4 var_namesr5r6rZ scale_varrZ argname_indexr,r,r-rsP       zOutScaleForInferencePass.applycCr rr,r#r,r,r-rr$z$OutScaleForInferencePass._scale_namerrr,r,r,r-rs  5rc@sHeZdZdZgdZdddddgddgddfd d Zd d Zd dZdS)AddQuantDequantPasszq Quantize the ops that do not have weights, and add quant_dequant op for the quantized ops's inputs. ZreluZrelu6Z leaky_relutanhZswishNr<r9r=elementwise_addpool2dc Cs||_t||_||_||_||_||_||_||_|jD]} | t t vs-J| dqdd|jD|_ |jdus@Jd|jdusIJddS)a? Constructor. Args: scope(static.Scope): The scope is used to initialize these new parameters. place(static.CPUPlace|static.CUDAPlace|str): place is used to initialize new parameters described above. If ``place`` is string, it can be It can be ``cpu`` or ``gpu:x``, where ``x`` is the index of the GPUs. moving_rate(float, optional): the param for 'quant_dequant_moving_average_abs_max' quantization. Default is 0.9. quant_bits(int, optional): quantization bit number for activation. Default is 8. skip_pattern(str, optional): The user-defined quantization skip pattern, which will be presented in the name scope of an op. When the skip pattern is detected in an op's name scope, the corresponding op will not be quantized. Default is 'skip_quant'. quantizable_op_type(list[str], optional): List the type of ops that will be quantized. Default is ["elementwise_add", "pool2d"]. rIcSrJrKr,rLr,r,r-rORrPz0AddQuantDequantPass.__init__..Nscope must not be None.place must not be None.) rQrrRr` _quant_bitsrerUr_quantizable_op_typerbrrc_quantizable_grad_op_type rjr)r*rorrprqrsrrr,r,r-ru(s"  zAddQuantDequantPass.__init__c sLt|ts Jd|jdur||_t}|}tt|ddd}|D] |j vrd}t|j t rL doJtfdd |j D}nt|j trf doe d|j d k} d ou d d k}|s|st|sq' d d  d|j ddt}d}|D] } d| vrd}nq|rq'|D]-} |j| } | |vr|| } n||| |j d\} } | || <|| | q|q'Wdn1swY|D]) |jvrD]} | |vr|j| } || }|| |qq||S) Add quant_dequant before some ops, such as the 'elementwise_add' and 'pool2d' op. Args: graph(IrGraph): the target graph. Returns: None rwN6Adding quant activation op:|{bar}| {n_fmt}/{total_fmt}rrFr}c3r~rrrrr,r-rurz,AddQuantDequantPass.apply..rrrr*rlrTquantized.dequantizedr) r!rrersrrrrrr$rrUrbrNrrrrrr7rrrr/r0r1/_insert_quant_dequant_moving_average_abs_max_oprrrrr)rjr3dequantized_vars_maprrr+ is_quantized arg_namesr=arg_namer6rr input_namerr,rr-rYs          G  zAddQuantDequantPass.applycCs|j|d|||d}|d}|tjkr%d}n |tjkr/d}nd}z+|jdurN||j vrNt j |j|g|d}n t j |j ||d}Wn t j tg|d}Y|j|dtjjjd g|d} t| ||j |j|| } || d } || d } |js|jtd tjjj|d gd } |tjkrd}n |tjkrd}nd}t| t jd g|d|j |j|jtdtjjj|d gd }t|t jd g|d|j |j|| }||}| | d<|| d<|| d<|| d<||j|j|d}|jd|| | d}||||| ||||||| |jsW|| |||||||||||| fS)z:Insert fake_quantize_dequantize_moving_average_abs_max op.z.quant_dequantrz.quant_dequant@scalerrrNrrrrzquant_dequant.staterzquant_dequant.accumrrrrrrr) rr$rrrrrrrrcr"rrQrr%rrrrrrr.rRrr#rer rrr`rr)rjr3r'rrrrrrrrrrrrrrrrr,r,r-rs             zCAddQuantDequantPass._insert_quant_dequant_moving_average_abs_max_op)r,r-r.r/_activation_typerurrr,r,r,r-rs 1 irc@sdeZdZdZ      ddd Zddejjjfd d Z d d Z ddZ ddZ ddZ ddZdS)InsertQuantizeLineara Insert quantize_linear and dequantize_linear op before ops. Args: place(paddle.CPUPlace|paddle.CUDAPlace|str): place is used to restore the weight tensors. If it's string, It can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. scope(paddle.Scope): scope is used to get the weight tensor values. quant_bits(int, optional): quantization bit number for weight. Default is 8. quant_axis(int, optional): quantization dimension of channels. When it is greater than or equal to 0, it will quantization with per channel, else quantization with per layer. Default is -1. channel_wise(bool, optional): Whether quantization with per channel or not. Default is False. moving_rate(float): the rate for 'moving average' method. is_test(bool, optional): Whether quantization with training or not. Default is True. scale_dict(dict, optional): calibration ranges of tensors output. r9rFr<TNc Cs4||_||_||_||_||_||_||_||_dSr)rRrQrr channel_wiserer`r) rjr*r)rrrrorsrr,r,r-ruGs  zInsertQuantizeLinear.__init__cCs|s J|d|s|n|}|j|||||d}|s|tjkr3d}n |tj kr=d}nd}| |}|j r]||j } t jjj} tj| |dt} nd} |} tjtg|d} |jdur||jvrtj|j|g|d} |j|| | g|d}t|| |j|jd} | dur|j||t jjj|t jjjd} t| tj|dd|j|j||d } | dur| | d <|j |jd }||d <d |i}|jst| |!}|jt"#dt jjj|dgd}|tjkr d}n |tj krd}nd}t|tjdg|d|j|j|jt"#dt jjj|dgd}t|tjdg|d|j|j| |!}| |!}||d<|| d<|| d<||d<||d<|j|d<|j$|d<|j%d|| |d}|&|||&||| dur|&| ||&|||js|&|||&|||&|||&|||&||||fS)Nrrrrrrrint32r  ZeroPointrrrYrrrrrrrrrsroquantize_linearr)'rr$rrrrrrrrrrrrrrrr"rrrrrcrr.rQrR_zero_point_nameINT32rrrerr#r rr`rr)rjr3r'r5rrrrrZscale_var_shapeZscale_var_typeZinit_scale_valuezero_point_noder1rrrrrrrrr,r,r-insert_quant_op[s                   z$InsertQuantizeLinear.insert_quant_opc Cs|s J|d|j|||||d}d}|durM|j||t j j j |t j j j d}t|tj|dd|j|j||d}|durZ||d<|j|jd}||d<|jd ||d |id } ||| ||| |dur||| || ||S) Nrrrrr rrrdequantize_linearrr)rr$rr rrrrrrrrrrr.r"rrQrRrrrr) rjr3r'rrrrr1rrr,r,r-insert_dequant_opsL       z&InsertQuantizeLinear.insert_dequant_opcCr r!r,r#r,r,r-r r$z(InsertQuantizeLinear._quantized_var_namecCr r%r,r#r,r,r-r ! r$z*InsertQuantizeLinear._dequantized_var_namecCr r'r,r#r,r,r-r' r$z*InsertQuantizeLinear._quantized_scale_namecCr r @zero_pointr,r#r,r,r-r- r$z%InsertQuantizeLinear._zero_point_name)r9rFr<TN)r,r-r.r/rurrrrrrrr rrr,r,r,r-r5s&  . rc@sreZdZdZdddddddddggddddddddfd d Zd d Zd dZddZddZddZ ddZ dS)QuantizationTransformPassV2z Quantize the ops that have weights. Add quant and dequant ops for the quantized ops's inputs. It is used in the new format of quantization. Nr9r:r;r<r=r>cCs||_t||_||_||_| |_| |_| |_| |_||_ ||_ ||_ d|_ gd}|dks2Jd||vr>t d|d||vrJt d|d||_||_||_||_| |_|jD]}|ttvslJ|d q\d d |jD|_||_d |_i|_i|_d S) a  Args: scope(paddle.Scope): When activation use 'range_abs_max' as the quantize type, this pass will create some new parameters. The scope is used to initialize these new parameters. place(paddle.CPUPlace|paddle.CUDAPlace|str): place is used to initialize new parameters described above. If it's string, It can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. weight_bits(int): quantization bit number for weights, the bias is not quantized. activation_bits(int): quantization bit number for activation. activation_quantize_type(str): quantization type for activation, now support 'abs_max', 'range_abs_max' and 'moving_average_abs_max'. If use 'abs_max' mode, the quantization scale will be calculated dynamically each step in both training and testing period. If use 'range_abs_max', a static quantization scale will be calculated during training and used in inference. weight_quantize_type(str): quantization type for weights, support 'abs_max' and 'channel_wise_abs_max'. The 'range_abs_max' usually is not used for weight, since weights are fixed once the model is well trained. window_size(int): the window size for 'range_abs_max' quantization. moving_rate(float): the param for 'moving_average_abs_max' quantization. skip_pattern(str or str list): The user-defined quantization skip pattern, which will be presented in the name scope of an op. When the skip pattern is detected in an op's name scope, the corresponding op will not be quantized. quantizable_op_type(list[str]): List the type of ops that will be quantized. Default is ["conv2d", "depthwise_conv2d", "mul"]. The quantizable_op_type in QuantizationFreezePass and ConvertToInt8Pass must be the same as this. weight_quantize_func(function): Function that defines how to quantize weight. Using this can quickly test if user's quantization method works or not. In this function, user should both define quantization function and dequantization function, that is, the function's input is non-quantized weight and function returns dequantized weight. If None, will use quantization op defined by 'weight_quantize_type'. Default is None. act_quantize_func(function): Function that defines how to quantize activation. Using this can quickly test if user's quantization method works or not. In this function, user should both define quantization and dequantization process, that is, the function's input is non-quantized activation and function returns dequantized activation. If None, will use quantization op defined by 'activation_quantize_type'. Default is None. weight_preprocess_func(function): Function that defines how to preprocess weight before quantization. Using this can quickly test if user's preprocess method works or not. The function's input is non-quantized weight and function returns processed weight to be quantized. If None, the weight will be quantized directly. Default is None. act_preprocess_func(function): Function that defines how to preprocess activation before quantization. Using this can quickly test if user's preprocess method works or not. The function's input is non-quantized activation and function returns processed activation to be quantized. If None, the activation will be quantized directly. Default is None. optimizer_func(function): Function return a optimizer. When 'is_test' is False and user want to use self-defined quantization function and preprocess function, this function must be set. Default is None. executor(paddle.Executor): If user want to use self-defined quantization function and preprocess function, executor must be set for initialization. Default is None. Examples: .. code-block:: python >>> # The original graph will be rewrite. >>> import paddle >>> import paddle.static as static >>> from paddle.static.quantization import QuantizationTransformPassV2 >>> from paddle.base.framework import IrGraph >>> from paddle.framework import core >>> graph = IrGraph(core.Graph(static.Program().desc), for_test=False) >>> place = paddle.CPUPlace() >>> scope = paddle.static.global_scope() >>> transform_pass = QuantizationTransformPassV2(scope, place) >>> transform_pass.apply(graph) Fr@rArDrErFrGrHrIcSrJrKr,rLr,r,r-rO rPz8QuantizationTransformPassV2.__init__..N)rQrrRrSrTrUrVrWrXrYrZr[_conv1dtranspose_flagr\r]r^r_r`rarbrrcrdrerfrgrhrir,r,r-ru: sP^      z$QuantizationTransformPassV2.__init__csd}t|jtrdotfdd|jD}nt|jtr6do5d|jdk}|rJ dd dddSdS) NFr}c3r~rrrrr,r-r rz@QuantizationTransformPassV2._quant_preprocess..rr=Trr)rjr4rr,rr-r s  z-QuantizationTransformPassV2._quant_preprocessc Cs8|dd|d}d}|j}|D]}||vr"q||jvr1|j|}n|}||jvr;q||jvsI||j vrKdnd} | r^|j dur^| ||j ||}n| sn|j durn| ||j ||}| r|j dur| ||j ||} |j|q| s|jdur| ||j||} |j|q||jvr|jn|j} | r|jn|j} d} d}| dkrd}|}|dko|d }|r|d n|}|jrd } d|_n |tjvrd nd } t|j|j| | ||j|jd }|j||||d\}}|||||}||j|<| r|}||||q|S)NrrrTFrrArrrrrrrrrorsr5r)rNrrr1r$rrrrpersistable_cast_output_varsrXrrYrVrrWrSrTr^r]rrrrrRrQr`rerrr)rjr3rNrrbr1r'rr$rrrrtrrrrinsert_quant_passrrr,r,r-r s           z.QuantizationTransformPassV2._transform_forwardcCsL|jD] }||vrq||jvr#|j|}||||qdSr)r1r$rrr)rjr3rNr'rr,r,r-rH s z/QuantizationTransformPassV2._transform_backwardcCsFd}|jD]}||vrq||jvs||jvr d}q|Sr)r1r$rrr)rjrNrr'r,r,r-rP s z'QuantizationTransformPassV2._has_weightc Csd|vsd|ddvrdS|dd}d}|D])}t|}||vrG||rG|dkrA|||s@|||}q|||}q|jD]}||krV|j n|j }||krb|j n|j } d} d} | dkrd } |tj vrxd nd} dtj vrtj d|||rdSt|j|j|| | |j|jd } ||kr|nd} | j|||| |d d \}} | ||| |d }||||qKdS)NrZ unsqueeze2FilterrrrFrATrrr)r5rr)r$r@rrrrrrr1rSrTr^r]rremoverrRrQr`rerrNrrr)rjr3rNZconv_weight_var_namerb_oprr'rrtrrrrrrr,r,r- _quant_conv1d\ s                z)QuantizationTransformPassV2._quant_conv1dcCst|ts Jd|jdur||_t|_g|_g|_dd| D|_| }g|_ | D]}| dkrP|j d |jvrP|j |jd q3|D]}| |jvsc| |jvrh||qSi|_tt|ddd 0}|D]%}| |jvr|||s||r|||n||||qxWdn1swY|D]}| |jvr||r|||q|S) rvrwNcSrxr,ryrzr,r,r-rO z5QuantizationTransformPassV2.apply..castrrrr)r!rrersrrrrrrrrr$r1rrrardrrrrrrrrrr)rjr3rrNrr,r,r-r sZ          z!QuantizationTransformPassV2.apply) r,r-r.r/rurrrrrrr,r,r,r-r4 s6 j Irc@s@eZdZdZgdZdddddgddgddfd d Zd d ZdS) AddQuantDequantPassV2z Quantize the ops that do not have weights, and add quant_linear and dequant_linear op for the quantized ops's inputs. It is used in the new format of quantization. rNr<r9r=rrc Cs||_t||_||_||_||_||_||_||_|jD]} | t t vs-J| dqdd|jD|_ |jdus@Jd|jdusIJdg|_ dS)aJ Args: scope(paddle.Scope): The scope is used to initialize these new parameters. place(paddle.CPUPlace|paddle.CUDAPlace|str): place is used to initialize new parameters described above. If ``place`` is string, it can be It can be ``cpu`` or ``gpu:x``, where ``x`` is the index of the GPUs. moving_rate(float, optional): the param for 'quant_dequant_moving_average_abs_max' quantization. Default is 0.9. quant_bits(int, optional): quantization bit number for activation. Default is 8. skip_pattern(str, optional): The user-defined quantization skip pattern, which will be presented in the name scope of an op. When the skip pattern is detected in an op's name scope, the corresponding op will not be quantized. Default is 'skip_quant'. quantizable_op_type(list[str], optional): List the type of ops that will be quantized. Default is ["elementwise_add", "pool2d"]. scale_dict(dict, optional): calibration ranges of tensors output. Examples: .. code-block:: python >>> # The original graph will be rewrite. >>> import paddle >>> import paddle.static as static >>> from paddle.static.quantization import AddQuantDequantPassV2 >>> from paddle.base.framework import IrGraph >>> from paddle.framework import core >>> graph = IrGraph(core.Graph(static.Program().desc), for_test=False) >>> place = paddle.CPUPlace() >>> scope = paddle.static.global_scope() >>> add_quant_dequant_pass = AddQuantDequantPassV2(scope, place) >>> add_quant_dequant_pass.apply(graph) rIcSrJrKr,rLr,r,r-rO- rPz2AddQuantDequantPassV2.__init__..Nrr)rQrrRr`rrerUrrrbrrcrrrr,r,r-ru s$,   zAddQuantDequantPassV2.__init__c st|ts Jd|jdur||_t}dd|D|_|}t t |ddd}|D]܉ |j vr d}t|j trWd oUtfd d |j D}nt|j trqd opd |j d k}d od dk}|s|rq1t}d}|D] } d| vrd}nq|rq1|D]i} |j| } | rq| tjtjtjfvrt d dnC| |vr|| } n2t!|j"|j#|j$d d|j%|j|j&d} | j'|| dd\} }| (|| |d} | || <|)| | q|*q1Wdn 1swY|D]* |j+vrI,D]}||vrG|j|} ||} |)| | q.q |S)rrwNcSrxr,ryrzr,r,r-rOF rz/AddQuantDequantPassV2.apply..rrrFr}c3r~rrrrr,r-rU rz.AddQuantDequantPassV2.apply..rrrrTz Since the zJ contains an input of type INT, the quantization of this layer is skipped.)rrrrorsrr)r)-r!rrersrrrrrrrr$rrUrbrNrrrrrrr/r0r1r2rrrrr_loggerwarningrrRrQrr`rrrrrrr)rjr3rrrr+rrr=rr6rrrrrr,rr-r5 s            \  zAddQuantDequantPassV2.apply)r,r-r.r/rrurr,r,r,r-r s Arc@s:eZdZdZdddZddZddZd d Zd d Zd S)ReplaceFakeQuantDequantPasszS replace quant-dequant ops with quantize_linear and dequantize_linear ops. r9cCs>t||_||_||_|jdusJd|jdusJddS)a Args: scope(paddle.Scope): The scope is used to initialize these new parameters. place(paddle.CPUPlace|paddle.CUDAPlace|str): place is used to initialize new parameters described above. If ``place`` is string, it can be It can be ``cpu`` or ``gpu:x``, where ``x`` is the index of the GPUs. quant_bits(int, optional): quantization bit number for activation. Default is 8. Examples: .. code-block:: python >>> # The original graph will be rewrite. >>> import paddle >>> import paddle.static as static >>> from paddle.static.quantization import ReplaceFakeQuantDequantPass >>> from paddle.base.framework import IrGraph >>> from paddle.framework import core >>> graph = IrGraph(core.Graph(static.Program().desc), for_test=False) >>> place = paddle.CPUPlace() >>> scope = paddle.static.global_scope() >>> replace_pass = ReplaceFakeQuantDequantPass(scope, place) >>> replace_pass.apply(graph) Nrr)rrRrQr)rjr)r*rr,r,r-ru s z$ReplaceFakeQuantDequantPass.__init__c Cst|ts Jdg}g}g}|D]}|dkr%||ddq|D]$}|tvs8|dkrN|dd}||vrI||q*||q*|D]'}||j |dd}||j |dd} | j D] } | | || qnqQ|D] }| ||| |q{||S)Nrwrrrr)r!rrr$rrB_fake_quant_dequant_op_listr@r0r1rr _replace_oprr) rjr3Zfake_quant_dequant_opsZremove_fake_quant_opsZobserver_out_node_namesrNr5rx_noderrr,r,r-r s>           z!ReplaceFakeQuantDequantPass.applycCs||j|dd}||j|dd}||j|dd}|dr2|dnd}|drB|dn|j}d|d>d}d|d} d} |} | dur|j | | t j jj|t j jjd } t| tj|d d |j|j|j|| |||d } |jd ||| |d ||| dd| id} ||| ||| | dur|| | || | |jd||| |d | || dd|id}|| ||||| dur|| ||||dS)Nrrrrrrrrrrrr)rrqminqmax)rr rrrr)r0r1r@rrBrNrrrrrr$rrrrrrr.r"rrQrRrrrrrr)rjr3rNrrrrrrrrZ quanted_noderrrr,r,r-r s           z'ReplaceFakeQuantDequantPass._replace_opcCr r!r,r#r,r,r-rR r$z/ReplaceFakeQuantDequantPass._quantized_var_namecCr rr,r#r,r,r-rX r$z,ReplaceFakeQuantDequantPass._zero_point_nameN)r9) r,r-r.r/rurrrrr,r,r,r-r s &Q rc@s@eZdZdZ   dddZddZd d Zd d Zd dZdS)QuantWeightPassa quant weights and remove weights input quantize_linear node. for example: `weight -> quant -> dequant -> conv2d` will be frozen into `weight -> dequant -> conv2d`, and weight will be scaled offline. Args: scope(paddle.Scope): scope is used to get the weight tensor values. place(paddle.CPUPlace|paddle.CUDAPlace|str): place is used to restore the weight tensors. If it's string, It can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. bias_correction(bool): whether use bias correction for post-training quantization. https://arxiv.org/abs/1810.05723. quant_bits(int, optional): quantization bit number for weight. Default is 8. save_int_weight(bool, optional): Whether the type saving the weight is int. Default is True. Examples: .. code-block:: python >>> # The original graph will be rewrite. >>> import paddle >>> import paddle.static as static >>> from paddle.static.quantization import QuantWeightPass >>> from paddle.base.framework import IrGraph >>> from paddle.framework import core >>> graph = IrGraph(core.Graph(paddle.static.Program().desc), for_test=False) >>> place = paddle.CPUPlace() >>> scope = paddle.static.global_scope() >>> quant_weight_pass = QuantWeightPass(scope, place) >>> quant_weight_pass.apply(graph) Fr9TcCsRt||_||_||_||_||_|jdusJd|jdus#Jdt|_dS)Nrr)rrRrQr2r_save_int_weightr&r;)rjr)r*r<rZsave_int_weightr,r,r-ru s  zQuantWeightPass.__init__cCst|ts Jdg}dd|D}|D]}||j|dd}|r||j|dd}||j|dd}||j|dd}| | } | j d vs\Jd | j d kre| d} | j d kru| d kru| d} n| } | | } |d} |d} | |jvr|j| tj| | | | dd} |jdurtj| | | | | d} |jr|jdkrtj}| |} || | |jD] }||||q||q||dS)NrwcSsg|] }|dkr|qS)rryrLr,r,r-rO sz)QuantWeightPass.apply..rrr rrr=r>rrr:rrT)Z onnx_formatr?r9) r!rrr0r1r@r2rrBrCr$rDrErFrNrr;rGrrHrIr2rJrrr"rr_rKrrrO)rjr3Zfake_quant_ops_for_weightZfake_quant_opsrrrrrrPrQrZ bits_lengthrRZsave_weight_dtyperr,r,r-r sx        zQuantWeightPass.applycrl)NcSrmr,rnror,r,r-rq rrz;QuantWeightPass._remove_unused_var_nodes..crsrrnrnrtr,r-ru rvz:QuantWeightPass._remove_unused_var_nodes..rwryr,rtr-rO r|z(QuantWeightPass._remove_unused_var_nodescCrdrrerfr,r,r-rC rgzQuantWeightPass._load_varcCrhrrirjr,r,r-rK rkzQuantWeightPass._restore_varN)Fr9T) r,r-r.r/rurrOrCrKr,r,r,r-r_ s# C rc@s:eZdZdZdgddfddZddZd d Zd d ZdS) AddQuantDequantForInferencePassz When export quant model, it will traverse to find the output of each op, and then insert the quant/dequant op after it. r9NTcCs8||_||_||_||_|r|ntt|_||_dS)a Args: scope(static.Scope): The scope is used to initialize these new parameters. place(paddle.CPUPlace|paddle.CUDAPlace|str): place is used to restore the weight tensors. If it's string, it can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. quant_bits(int, optional): quantization bit number for weight. Default is 8. N) rQrRr_only_observerrbrrcr_calibration_range_dict)rjr)r*rrqZcalibration_range_dict only_observerr,r,r-ru s  z(AddQuantDequantForInferencePass.__init__c Cs t|ts Jdi}t}|D]>}||jvrQt|}|D]-}| |j |}| t j t jt jfvr9q#||vrB||}n |||}|||<|||<q#q|D]-}|dkrd||qVt|}|D]}||r| |j|} || |||qkqV|S)E Args: graph(IrGraph): the target graph. rwr)r!rrrrr$rrrr0rrrrrr_insert_quant_dequant_oprr/getr1r) rjr3Zdequant_node_maprr4rr5rrr6r,r,r-r sR            z%AddQuantDequantForInferencePass.applycCr rr,r#r,r,r-r> r$z+AddQuantDequantForInferencePass._scale_namec Cs|s J|d|}d}|j|d|||d}z ||||}WnH|j rn||j vrn|j |}|j |||dg|d}|t j kr]dnd}t |tj||d|j|jn td |d YdSYz|||d } Wn*|j |d tjjj|tjjjd} t | tj|d d|j|jY||d } | dur| | d<||j|jd} tjjj| d<d|i} |jd| | | d} | || | || | dur| | | | | ||j|d|||d}||d } | dur| | d<d|j|jd} tjjj| d<|jd| | d|id}| ||| ||| durM| | || |||S)Nrrr"rrrrrzCannot find the target node z% in scope, so skip adding quant node.rrr r)rrrrrrrr&r)!rr$rrrrr0rrrrrrr.r"rrQrRrrrrrrrrrrrrrrr)rjr3r'r5rrrrrrr1rrrrrr,r,r-rD s                    z8AddQuantDequantForInferencePass._insert_quant_dequant_op)r,r-r.r/rurrrr,r,r,r-r s 1 rc@s>eZdZdZ  dddZddZdd Zd d Zd d ZdS)AddQuantDequantForResidualz_ Quantize the residual connections. Add quant and dequant ops for the residual inputs. r9TcCsDt||_||_||_||_|jdusJd|jdus JddS)a Args: scope(static.Scope): The scope is used to initialize these new parameters. place(paddle.CPUPlace|paddle.CUDAPlace|str): place is used to restore the weight tensors. If it's string, it can be ``cpu``, and ``gpu:x``, where ``x`` is the index of the GPUs. quant_bits(int, optional): quantization bit number for weight. Default is 8. is_test(bool, optional): Whether quantization with training or not. Default is True. Nrr)rrRrQrre)rjr)r*rrsr,r,r-ru s z#AddQuantDequantForResidual.__init__cCst|ts Jd||}||}|D]9}|dkr q|jd}|jd}||vs6||vr7q||||krD|jdn|jd}||||qdS)rrwrrrN)r!r_all_weight_node_names_var_name_orderrr$r1_insert_quant_dequant)rjr3weight_var_namesvar_node_names_with_orderrNZfirst_input_nameZsecond_input_nameZ skip_noder,r,r-r s*      z AddQuantDequantForResidual.applycCsRdd|D}|D]}|dkr&|jdr&||jdq |S)zM Return a list of weight variables (including casted weight) cSrxr,ry)rMrr,r,r-rO rzEAddQuantDequantForResidual._all_weight_node_names..rr)rrr$r1r2rr)rjr3rrNr,r,r-r s z1AddQuantDequantForResidual._all_weight_node_namescCsL|}i}t|D]\}}|jD]}|}||dur"|||<qq |S)zY Return a dictionary with variable names as key and their order as value N)Z topology_sort enumerater1r$r)rjr3Z ordered_opsridxr4Z in_var_nodeZ in_var_namer,r,r-rs z*AddQuantDequantForResidual._var_name_orderc Csnt|j|j|jdd|jd}|d}|d}|j||||d\}}| ||||} | || |dS)zf Insert per tensor quantize_linear and dequantize_linear node between var_node and op rF)rrrrsz.skiprrN) rrRrQrrer$rNrrrr) rjr3r'rNrZquant_var_namerrrrr,r,r-rs( z0AddQuantDequantForResidual._insert_quant_dequantN)r9T) r,r-r.r/rurrrrr,r,r,r-r s  r)2rloggingnumpyr"rrrZbase.frameworkrrZ frameworkrrZstaticr r r r r Z log_helperrrZ quant_configrrrr,INFOrr4r6rZ _conv_opsrr.r7r8r0r~rrrrrrrrrrrr,r,r,r-s|     eo\*.H:M)Q