o * i@sddlmZddlZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddl mZmZddlmZmZddlmZdd lmZmZmZdd lmZd d lmZerpdd l mZddlmZddl m!Z!Gddde Z"GdddeZ#ddZ$GdddZ%Gddde%Z&dS)) annotationsN) defaultdict)Enum) TYPE_CHECKINGAny TypedDict)_C_ops _legacy_C_ops)core unique_name) check_type)Operator_dygraph_tracer in_pir_mode)in_dynamic_mode)amp_global_state)Tensor)OptimizerWithMixedPrecision) Optimizerc@sNeZdZUded<ded<ded<ded<ded<ded <ded <d ed <d S)_ScaleStateDictrscalefloat incr_ratio decr_ratiointincr_every_n_stepsdecr_every_n_nan_or_inf incr_count decr_countbooluse_dynamic_loss_scalingN)__name__ __module__ __qualname____annotations__r&r&b/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/paddle/amp/grad_scaler.pyr)s  rc@seZdZdZdZdZdS)OptimizerStaterrN)r"r#r$INITUNSCALEDSTEPPEDr&r&r&r'r(4sr(cCs dtjiS)Nstate)r(r*r&r&r&r'_refresh_optimizer_state:s r.c@seZdZdZ       dJdKddZdLddZdMd d!Zd"d#Zd$d%ZdNd&d'Z dNd(d)Z dOd*d+Z dPd-d.Z dOd/d0Z dQd2d3ZdOd4d5ZdRd7d8ZdSd9d:ZdTdd?ZdUdAdBZdVdDdEZdWdGdHZdIS)X AmpScalera AmpScaler is used for Auto-Mixed-Precision training/inferring in imperative mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has seventeen methods `scale()`, `unscale_()`, `minimize()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling. Commonly, it is used together with `amp_guard` to achieve Auto-Mixed-Precision in imperative mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 2**15. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 1000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 2. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. Returns: An AmpScaler object. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) T@@?renabler init_loss_scalingrrrrrrr!returnNonec Cstr)t}|s td|r)|js)|js)|js)td|jdd}||_ d|_ d|_ d|_ |j r|dks@Jd|dksHJd||_ ||_ ||_||_||_d|_d|_||_ tr|tjjjd d gtd tjjj|j d d |_ dSttdg tj!|_"ttdg tj!|_#ttdg tj!|_$ttdg tj!|_%ttdg tj!|_&tt|j g tj'|_ d|_(t)t*|_+dSdS)Nz;current_tracer is None, maybe it is not in imperative mode.zWAmpScaler can only be enabled on CUDAPlace, XPUPlace and CustomPlace, current place is z, so it makes no effect.F?zThe incr_ratio must be > 1.0.zThe decr_ratio must be < 1.0.rfloat32rZ loss_scaling)value)dtypeshapename initializer),rr ValueErrorZ_expected_placeZ is_gpu_placeZ is_xpu_placeZis_custom_placewarningswarn_enable_use_dynamic_loss_scaling_init_loss_scaling_scale _incr_ratio _decr_ratio_incr_every_n_steps_decr_every_n_nan_or_inf _incr_count _decr_countrpaddlepirr Zcreate_persistable_valuer generatennr>ZConstantInitializer to_tensornparrayastypeZbool_ _found_inf_temp_found_inf_value_false_temp_found_inf_fp16_temp_found_inf_bf16_temp_found_inf_fp32r9_cache_found_infrr._optimizer_states) selfr4r5rrrrr!Ztracerr&r&r'__init__os|    zAmpScaler.__init__varrcCst|dtjtjjfd|jr,tjdkr,|jr,d|_d|_d|_ t dtjdt rn|j tjjkr;|d}|js@|Stj||j}|}|}|jrl|jrltjjj|jj|j|j|jj|_|S|jru|sw|S||jS) al Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`AmpScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The Tensor to scale. Returns: The scaled Tensor or original Tensor. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) r]zAmpScaler.scale()float16Fr8z6It is not recommended to use dynamic loss scaling for z&, so GradScaler is disable by default.r9)r rLrrMValuerBrZ amp_dtyperCrDr@rArr;r ZDataTypeZFLOAT32rSrmultiplyrEZget_defining_opZ dist_attrbaseZ libpaddleZcreate_op_dist_attributeZ process_meshZoperandsresultsZchunk_idZ_is_initialized)r[r]Z scale_outZ multiply_opZ src_var_opr&r&r'rsJ      zAmpScaler.scale optimizer'Optimizer | OptimizerWithMixedPrecisionargsrkwargs2tuple[list[Operator], list[tuple[Tensor, Tensor]]]cOsFtr@t|tjjjjsJ|j|_|j|_|j |_ |d|_ |jr8|j |_ |j |_ |j|_|j|_d|_d|_|j|i|S|jsK|j|i|S|jt|}|dtjur^||d\}}t|dr|d|j|j|i|\}}|d|_n|jrd|_n |j|i|\}}d|_|jr|tt |_||fS) a This function is similar as `Optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `Optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `Optimizer.minimize()`. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, ... parameters=model.parameters() ... ) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) rNr-)NN_set_auxiliary_var found_infTF)!r isinstancerLZstaticamp decoratorrrCrDrEZ _loss_scalingZ _scaled_lossrHrIrFrGZ_num_good_stepsZ_num_bad_stepsminimizerBrZidr(r*_unscalehasattrrhrT_get_auxiliary_varrY_updaterr.)r[rcrerfoptimizer_stateZ optimize_opsZ params_gradsr&r&r'rmsH*     zAmpScaler.minimizec Cs|jsdS|jt|}|dtjurtd|dtjur"tdt|ddrzt|j dt rzg}g}g}g}|j D]=}|dD]6}| durw| | | j tjkr`| | qA| j tjkrp| | qA| | qAq;n+trtj|j\}}}ndd |jD}d d |D}d d |D}d d |D}|j|_t|rt||j||jt|j|j|_t|rt||j||jt|j|j|_t|rt||j||j t|j|j |_tj|d<dS) a Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Nr-zMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step(). _param_groupsrparamscSs g|] }|dur|qSN) _grad_ivar.0paramr&r&r' s  z&AmpScaler._unscale..cSg|] }|jtjkr|qSr&)r;rLr^rxr&r&r'r{  cSr|r&)r;rLbfloat16rxr&r&r'r{r}cSr|r&)r;rLr9rxr&r&r'r{r})!rBrZrnr(r+ RuntimeErrorr,getattrrjrtdictrwappendr;rLr^r~rr eagerZget_grads_listsZ_parameter_listrUrTlenr Zcheck_finite_and_unscalerErVrZ bitwise_orrWrX) r[rcrsZ param_gradsZparam_grads_fp16Zparam_grads_bf16Zparam_grads_fp32grouprzr&r&r'robs      zAmpScaler._unscalecCs|jsdS|jr9d|_|jd|_|j|jkr7tdt|jdt|jdt|j|j|j|_d|_dSd|_|jd|_|j|j krR|j|j |_d|_dS)z+ Updates the loss_scaling. Nrrz$Found inf or nan, current scale is: z, decrease to: *) rBrYrJrKrIprintrrErGrHrFr[r&r&r'rrs$  &  zAmpScaler._updatecC|jS)z Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. )rBrr&r&r' is_enablezAmpScaler.is_enablecCr)z Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamically return true. )rCrr&r&r'is_use_dynamic_loss_scalingrz%AmpScaler.is_use_dynamic_loss_scalingcCr)z Return the initial loss scaling factor. Returns: float: the initial loss scaling factor. )rDrr&r&r'get_init_loss_scalingrzAmpScaler.get_init_loss_scalingnew_init_loss_scalingcCs(||_tt|jgtj|_dS)z Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(int): The new_init_loss_scaling used to update initial loss scaling factor.s N)rDrLrPrQrRrSr9rEr[rr&r&r'set_init_loss_scalings zAmpScaler.set_init_loss_scalingcCr)z Return the multiplier to use when increasing the loss scaling. Returns: float: the multiplier to use when increasing the loss scaling. rFrr&r&r'get_incr_ratio rzAmpScaler.get_incr_rationew_incr_ratiocCs|dksJd||_dS)a  Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. r8z!The new_incr_ratio must be > 1.0.Nrr[rr&r&r'set_incr_ratio zAmpScaler.set_incr_ratiocCr)z Get the less-than-one-multiplier to use when decreasing the loss scaling. Returns: float: the less-than-one-multiplier to use when decreasing the loss scaling. rGrr&r&r'get_decr_ratiorzAmpScaler.get_decr_rationew_decr_ratiocCs|dksJd||_dS)a) Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. r8z!The new_decr_ratio must be < 1.0.Nrr[rr&r&r'set_decr_ratio&rzAmpScaler.set_decr_ratiocCr)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Returns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. rHrr&r&r'get_incr_every_n_steps0rz AmpScaler.get_incr_every_n_stepsnew_incr_every_n_stepscC ||_dS)a^ Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Nrr[rr&r&r'set_incr_every_n_steps9s z AmpScaler.set_incr_every_n_stepscCr)a Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Returns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. rIrr&r&r'get_decr_every_n_nan_or_infBrz%AmpScaler.get_decr_every_n_nan_or_infnew_decr_every_n_nan_or_infcCr)au Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Nrr[rr&r&r'set_decr_every_n_nan_or_infKs z%AmpScaler.set_decr_every_n_nan_or_infrc Cs4|jr|j|j|j|j|j|j|j|j dSiS)a Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. )rrrrrrrr!) rBrEnumpyrFrGrHrIrJrKrCrr&r&r' state_dictVs zAmpScaler.state_dictrcCs|jsdSt|dkrtd|dd|_tt|jgtj |_ |d|_ |d|_ |d|_ |d|_|d |_|d |_|d |_dS) z Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `AmpScaler.state_dict()`. NrzdThe input state dict is empty, possibly because it was saved from a disabled instance of GradScaler.rrrrrrrr!)rBrrrDrLrPrQrRrSr9rErFrGrHrIrJrKrCr[rr&r&r'load_state_dictts"       zAmpScaler.load_state_dictN)Tr0r1r2r3rTr4r r5rrrrrrrrrr!r r6r7r]rr6rrcrdrerrfrr6rgr6r r6rrrr6r7rrr6r7rrr6r7r6rrrr6r7rrr6r7r6rrrr6r7)r"r#r$__doc__r\rrmrorrrrrrrrrrrrrrrrr&r&r&r'r/>s82 L J]h  r/cs2eZdZdZ       dLdMfdd ZdNfdd ZdOfd d! ZdPd#d$ZdQd%d&Zfd'd(Z dRfd)d* Z dRfd+d, Z dSfd-d. Z dTfd0d1 Z dSfd2d3 ZdUfd5d6 ZdSfd7d8 ZdVfd:d; ZdWfd>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() T@r1r2rr4r r5rrrrrrr!r6r7c st|||||||dSrv)superr\)r[r4r5rrrrr! __class__r&r'r\s zGradScaler.__init__r]rc t|S)aJ Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The tensor to scale. Returns: The scaled tensor or original tensor. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rr)r[r]rr&r'rs zGradScaler.scalercrdrerrfrgcstj|g|Ri|S)a This function is similar as `optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `optimizer.minimize()`. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rrm)r[rcrerfrr&r'rms'zGradScaler.minimizercCs|js|S|jt|}|dtjurtd|dtjur%||t |dr<| d|j || d|_ n|j rCd|_ n|d|_ tj|d<|jsYtt|_dSdS)at This function is similar as `optimizer.step()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Args: optimizer(Optimizer): The optimizer used to update parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() r-z7step() has already been called since the last update().rhriTFN)rBsteprZrnr(r,rr*rorprhrTrqrYrCrr.)r[rcrsr&r&r'r"s*   zGradScaler.stepcCs&|jsdS|jr|tt|_dS)a Updates the loss_scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() N)rBrCrrrr.rZrr&r&r'update^s  zGradScaler.updatecr)aE Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.unscale_(optimizer) # unscale the parameter >>> scaler.step(optimizer) >>> scaler.update() >>> optimizer.clear_grad() )rro)r[rcrr&r'unscale_~s !zGradScaler.unscale_c tS)a Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> enable = scaler.is_enable() >>> print(enable) True )rrrrr&r'r zGradScaler.is_enablecr)ax Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamically return True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> use_dynamic_loss_scaling = scaler.is_use_dynamic_loss_scaling() >>> print(use_dynamic_loss_scaling) True )rrrrr&r'rrz&GradScaler.is_use_dynamic_loss_scalingcr)a& Return the initial loss scaling factor. Returns: float: the initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> init_loss_scaling = scaler.get_init_loss_scaling() >>> print(init_loss_scaling) 1024 )rrrrr&r'rrz GradScaler.get_init_loss_scalingrct|dS)a Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(float): The new_init_loss_scaling used to update initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_init_loss_scaling()) 1024 >>> new_init_loss_scaling = 1000 >>> scaler.set_init_loss_scaling(new_init_loss_scaling) >>> print(scaler.get_init_loss_scaling()) 1000 N)rrrrr&r'rz GradScaler.set_init_loss_scalingcr)a> Return the multiplier to use when increasing the loss scaling. Returns: float: the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_ratio = scaler.get_incr_ratio() >>> print(incr_ratio) 2.0 )rrrrr&r'rrzGradScaler.get_incr_ratiorcr)a Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_ratio()) 2.0 >>> new_incr_ratio = 3.0 >>> scaler.set_incr_ratio(new_incr_ratio) >>> print(scaler.get_incr_ratio()) 3.0 N)rrrrr&r'r+rzGradScaler.set_incr_ratiocr)aW Get the less-than-one-multiplier to use when decreasing the loss scaling. Returns: float: the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_ratio = scaler.get_decr_ratio() >>> print(decr_ratio) 0.5 )rrrrr&r'rIrzGradScaler.get_decr_ratiorcr)a7 Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_ratio()) 0.5 >>> new_decr_ratio = 0.1 >>> scaler.set_decr_ratio(new_decr_ratio) >>> print(scaler.get_decr_ratio()) 0.1 N)rrrrr&r'rdrzGradScaler.set_decr_ratiocr)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Returns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_every_n_steps = scaler.get_incr_every_n_steps() >>> print(incr_every_n_steps) 1000 )rrrrr&r'rrz!GradScaler.get_incr_every_n_stepsrcr)a Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_every_n_steps()) 1000 >>> new_incr_every_n_steps = 2000 >>> scaler.set_incr_every_n_steps(new_incr_every_n_steps) >>> print(scaler.get_incr_every_n_steps()) 2000 N)rrrrr&r'rrz!GradScaler.set_incr_every_n_stepscr)a Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Returns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_every_n_nan_or_inf = scaler.get_decr_every_n_nan_or_inf() >>> print(decr_every_n_nan_or_inf) 2 )rrrrr&r'rrz&GradScaler.get_decr_every_n_nan_or_infrcr)a Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_every_n_nan_or_inf()) 2 >>> new_decr_every_n_nan_or_inf = 3 >>> scaler.set_decr_every_n_nan_or_inf(new_decr_every_n_nan_or_inf) >>> print(scaler.get_decr_every_n_nan_or_inf()) 3 N)rrrrr&r'rsz&GradScaler.set_decr_every_n_nan_or_infrcr)a0 Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() )rrrrr&r'rs "zGradScaler.state_dictrcr)a: Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `GradScaler.state_dict()`. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() >>> scaler.load_state_dict(scaler_state) N)rrrrr&r'rszGradScaler.load_state_dict)Trr1r2rrTrrr)rcrr6r7)r6r7rrrrrrrrrr)r"r#r$rr\rrmrrrrrrrrrrrrrrrrr __classcell__r&r&rr'rs:5 ) < # $r)' __future__rr@ collectionsrenumrtypingrrrrrQrLrr Z paddle.baser r Zpaddle.base.data_feederr Zpaddle.base.frameworkr rrZpaddle.frameworkrZ auto_castrrZpaddle.static.amp.decoratorrZ!python.paddle.optimizer.optimizerrrr(r.r/rr&r&r&r's4          W