o + i'y@sddlmZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m Z ddlZddlmZd d lmZe rZdd lmZdd lmZdd lmZddlmZd dlmZgZGddde ZGddde ZddZddZd"ddZ    d#ddZ!Gd d!d!eZ"dS)$) annotationsN) defaultdict)reduce) TYPE_CHECKINGNoReturn TypedDict) NotRequired) framework) Optimizer)Sequence)Tensor)GradientClipBase)WeightDecayRegularizer)_ParameterConfigc@sfeZdZUded<ded<ded<ded<ded<ded <ded <ded <ded <d ed<ded<dS) _LbfgsStateint func_evalsn_iterrdalphaz list[Tensor]old_ykold_skroH_diagprev_flat_gradfloat prev_losszNotRequired[list[Tensor]]alN__name__ __module__ __qualname____annotations__r%r%b/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/paddle/optimizer/lbfgs.pyr*s  rc@seZdZUded<dS)_LbfgsStateDictrstateNr r%r%r%r&r'8s r'cCs.tjrtddkrtddSdSdS)z-Check and warn about TF32 acceleration statusZNVIDIA_TF32_OVERRIDE0zWarning! TF32 Tensor Cores are enabled by default on some NVIDIA GPUs for faster computation, but may compromise numerical precision in specific cases, particularly with the L-BFGS optimizer.To disable it, set: NVIDIA_TF32_OVERRIDE=0N)paddleZdeviceZis_compiled_with_cudaosgetenvwarningswarnr%r%r%r&check_tf32_override<sr/cCs||jddS)z NOTE: This is a temporary workaround for unstable result computed by `paddle.dot`, which will be reverted when the problem is fixed." Zaxis)sumxyr%r%r&dotIsr6c Cs|dur |\}}n||kr||fn||f\}}||d||||} | d||} | dkrj| } ||krN||||| | ||d| } n||||| | ||d| } tt| ||S||dS)a]Cubic interpolation between (x1, f1, g1) and (x2, f2, g2). Use two points and their gradient to determine a cubic function and get the minimum point between them in the cubic curve. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp59: formula 3.59 Args: x1, f1, g1: point1's position, value and gradient. x2, f2, g2: point2's position, value and gradient. bounds: bounds of interpolation area Returns: min_pos: the minimum point between the specified points in the cubic curve. Nr rg@)sqrtminmax) x1f1g1Zx2f2g2boundsZ xmin_boundZ xmax_boundZd1Z d2_squareZd2Zmin_posr%r%r&_cubic_interpolateQs *( rA-C6??& .>c ! Cs|} |}||||\} } d}t| |}d|||f\}}}}d}d}|| kr| ||||ks=|dkrP| |krP||g}|| g}|| g}||g}njt|| |kre|g}| g}| g}d}nU|dkr|||g}|| g}|| g}||g}n>|d||}|d}|}t||||| |||fd}|}| }| }|}||||\} } |d7}t| |}|d7}|| ks+|| krd|g}|| g}|| g}d}|d|dkrd nd \}}|s|| krt|d|d| | krnt|d|d|d|d|d|d}d t|t|} tt|||t|| krU|s1|t|ks1|t|krRt|t|t|t|krIt|| }nt|| }d}nd}nd}||||\} } |d7}t| |}|d7}| ||||ks~| ||kr|||<| ||<| ||<|||<|d|dkrd nd \}}nDt|| |krd}n%|||||dkr||||<||||<||||<||||<|||<| ||<| ||<|||<|s|| ks||}||} ||} | | ||fS) ag Implements of line search algorithm that satisfies the strong Wolfe conditions using double zoom. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp60: Algorithm 3.5 (Line Search Algorithm). Args: obj_func: the objective function to minimize. ```` accepts a multivariate input and returns a scalar. xk (Tensor): the starting point of the iterates. alpha (Scalar): the initial step size. d (Tensor): search direction. loss (scalar): the initial loss grad (Tensor): the initial grad c1 (Scalar): parameter for sufficient decrease condition. c2 (Scalar): parameter for curvature condition. tolerance_change (Scalar): terminates if the change of function value/position/parameter between two iterations is smaller than this value. max_ls(int): max iteration of line search. alpha_max (float): max step length. Returns: loss_new (Scaler): loss of obj_func at final alpha. grad_new, (Tensor): derivative of obj_func at final alpha. alpha(Tensor): optimal step length, or 0. if the line search algorithm did not converge. ls_func_evals (Scaler): number of objective function called in line search process. Following summarizes the essentials of the strong Wolfe line search algorithm. Some notations used in the description: - `func` denotes the objective function. - `obi_func` is a function of step size alpha, restricting `obj_func` on a line. obi_func = func(xk + alpha * d), where xk is the position of k'th iterate, d is the line search direction(decent direction), and a is the step size. - alpha : substitute of alpha - a1 is alpha of last iteration, which is alpha_(i-1). - a2 is alpha of current iteration, which is alpha_i. - a_lo is alpha in left position when calls zoom, which is alpha_low. - a_hi is alpha in right position when calls zoom, which is alpha_high. Line Search Algorithm: repeat Compute obi_func(a2) and derphi(a2). 1. If obi_func(a2) > obi_func(0) + c_1 * a2 * obi_func'(0) or [obi_func(a2) >= obi_func(a1) and i > 1], alpha= zoom(a1, a2) and stop; 2. If |obi_func'(a2)| <= -c_2 * obi_func'(0), alpha= a2 and stop; 3. If obi_func'(a2) >= 0, alpha= zoom(a2, a1) and stop; a1 = a2 a2 = min(2 * a2, a2) i = i + 1 end(repeat) zoom(a_lo, a_hi) Algorithm: repeat aj = cubic_interpolation(a_lo, a_hi) Compute obi_func(aj) and derphi(aj). 1. If obi_func(aj) > obi_func(0) + c_1 * aj * obi_func'(0) or obi_func(aj) >= obi_func(a_lo), then a_hi <- aj; 2. 2.1. If |obi_func'(aj)| <= -c_2 * obi_func'(0), then alpha= a2 and stop; 2.2. If obi_func'(aj) * (a2 - a1) >= 0, then a_hi = a_lo a_lo = aj; end(repeat) reference: https://github.com/pytorch/pytorch r rFTg{Gz? )r@r0)rr )r rg?)absr:cloner6rAr9)!obj_funcZxkrrlossgradgtdc1c2tolerance_changeZmax_lsZd_normZloss_newZgrad_new ls_func_evalsZgtd_newZt_prevZf_prevZg_prevZgtd_prevdoneZls_iterZbracketZ bracket_fZ bracket_gZ bracket_gtdZmin_stepZmax_steptmpZinsuf_progressZlow_posZhigh_posZepsr%r%r& _strong_wolfeus X     3 ""        I rScseZdZdZ           d3d4fdd Zd5dd Zd6d!d"Zd#d$Zd%d&Zd'd(Z d)d*Z d+d,Z e j d7d.d/Z d8d9d1d2ZZS):LBFGSaJ The L-BFGS is a quasi-Newton method for solving an unconstrained optimization problem over a differentiable function. Closely related is the Newton method for minimization. Consider the iterate update formula: .. math:: x_{k+1} = x_{k} + H_k \nabla{f_k} If :math:`H_k` is the inverse Hessian of :math:`f` at :math:`x_k`, then it's the Newton method. If :math:`H_k` is symmetric and positive definite, used as an approximation of the inverse Hessian, then it's a quasi-Newton. In practice, the approximated Hessians are obtained by only using the gradients, over either whole or part of the search history, the former is BFGS, the latter is L-BFGS. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp179: Algorithm 7.5 (L-BFGS). Args: learning_rate (float, optional): learning rate .The default value is 1. max_iter (int, optional): maximal number of iterations per optimization step. The default value is 20. max_eval (int|None, optional): maximal number of function evaluations per optimization step. The default value is max_iter * 1.25. tolerance_grad (float, optional): termination tolerance on first order optimality The default value is 1e-5. tolerance_change (float, optional): termination tolerance on function value/parameter changes. The default value is 1e-9. history_size (int, optional): update history size. The default value is 100. line_search_fn (string|None, optional): either 'strong_wolfe' or None. The default value is strong_wolfe. parameters (list|tuple|None, optional): List/Tuple of ``Tensor`` names to update to minimize ``loss``. \ This parameter is required in dygraph mode. The default value is None. weight_decay (int|float|WeightDecayRegularizer|None, optional): The strategy of regularization. \ It can be a int or float value as coeff of L2 regularization or \ :ref:`api_paddle_regularizer_L1Decay`, :ref:`api_paddle_regularizer_L2Decay`. If a parameter has set regularizer using :ref:`api_paddle_ParamAttr` already, \ the regularization setting here in optimizer will be ignored for this parameter. \ Otherwise, the regularization setting here in optimizer will take effect. \ Default None, meaning there is no regularization. grad_clip (GradientClipBase|None, optional): Gradient clipping strategy, it's an instance of \ some derived class of ``GradientClipBase`` . There are three clipping strategies \ ( :ref:`api_paddle_nn_ClipGradByGlobalNorm` , :ref:`api_paddle_nn_ClipGradByNorm` , \ :ref:`api_paddle_nn_ClipGradByValue` ). Default None, meaning there is no gradient clipping. name (str|None, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. Return: loss (Tensor): the final loss of closure. Examples: .. code-block:: python >>> import paddle >>> import numpy as np >>> paddle.disable_static() >>> np.random.seed(0) >>> np_w = np.random.rand(1).astype(np.float32) >>> np_x = np.random.rand(1).astype(np.float32) >>> inputs = [np.random.rand(1).astype(np.float32) for i in range(10)] >>> # y = 2x >>> targets = [2 * x for x in inputs] >>> class Net(paddle.nn.Layer): ... def __init__(self): ... super().__init__() ... w = paddle.to_tensor(np_w) ... self.w = paddle.create_parameter(shape=w.shape, dtype=w.dtype, default_initializer=paddle.nn.initializer.Assign(w)) ... ... def forward(self, x): ... return self.w * x ... >>> net = Net() >>> opt = paddle.optimizer.LBFGS(learning_rate=1, max_iter=1, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn='strong_wolfe', parameters=net.parameters()) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print('loss: ', loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... opt.step(closure) ... >>> for input_np, target_np in zip(inputs, targets): ... input = paddle.to_tensor(input_np) ... target = paddle.to_tensor(target_np) ... train_step(input, target) ?NHz>rDd learning_ratermax_iterrmax_eval int | Nonetolerance_gradrO history_sizeline_search_fn str | None parameters4Sequence[Tensor] | Sequence[_ParameterConfig] | None weight_decay%float | WeightDecayRegularizer | None grad_clipGradientClipBase | NonenamereturnNonec st|dur |dd}||_||_||_||_||_||_||_t|t j r0t dt |t t|_tjd|| | | dt|jdtsM|j|_nt|jD] \} } | d|_qRd|_dS)Nz^parameters argument given to the optimizer should be an iterable of Tensors or dicts, but got rU)rYrarcrergrparams)r/rYrZr[r]rOr^r_ isinstancer*r TypeErrortyperdictr(super__init__Z_parameter_list_params enumerateZ _param_groups _numel_cache)selfrYrZr[r]rOr^r_rarcrergidxZ param_group __class__r%r&rrs:      zLBFGS.__init__r'cCs.i}|jD] \}}|||iqd|iS)apReturns the state of the optimizer as a :class:`dict`. Return: state, a dict holding current optimization state. Its content differs between optimizer classes. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... opt.clear_grad() ... loss.backward() ... return loss ... ... opt.step(closure) ... >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> n_iter = 0 >>> while n_iter < 20: ... loss = train_step(inputs, targets) ... n_iter = opt.state_dict()["state"]["func_evals"] ... print("n_iter:", n_iter) r()r(itemsupdate)rvZ packed_statekvr%r%r& state_dicts.zLBFGS.state_dictcCs$|jdurtdd|jd|_|jS)NcSs ||SN)numel)totalpr%r%r&+s zLBFGS._numel..r)rurrsrvr%r%r&_numel's  z LBFGS._numelcCsTg}|jD]}|jdurt|dg}n|jdg}||qtj|ddS)Nr0rr1)rsrKr*Z zeros_likereshapeappendconcat)rvZviewsrviewr%r%r&_gather_flat_grad0s   zLBFGS._gather_flat_gradc Cstd}|jD]*}|jgkrtdd|jnd}t||||||j||}||7}q||ks8JdS)NrcSs||Srr%r3r%r%r&r>sz!LBFGS._add_grad..r )rsshaperr*assignaddrr)rvr directionoffsetrrr%r%r& _add_grad;s  zLBFGS._add_gradcCsdd|jDS)NcSsg|]}|qSr%)rH).0rr%r%r& Isz&LBFGS._clone_param..)rsrr%r%r& _clone_paramHszLBFGS._clone_paramcCs&t|j|D] \}}t||qdSr)ziprsr*r)rvZ params_datarZpdatar%r%r& _set_paramKszLBFGS._set_paramcCs0|||t|}|}||||fSr)rrrr)rvclosurer4rrrJ flat_gradr%r%r&_directional_evaluateOs   zLBFGS._directional_evaluaterc% stXtj}j}j}j}j}j}j }j } | dd| dd} t | } d} | dd7< } | |k}|rX| WdS| d}| d}| d}| d }| d }| d }| d }| d }d}||kr5|d7}| dd7<| ddkr| }g}g}g}tjd| jd}n| |}|tj||jd}t||}|dkrt||kr|d|d|d|||||d||t||}t|}d| vrdg|| d<| d}| }t|dddD] }t|||||||<t||||| |qt||}}t|D]}t|||||} t|||||| |q?|duri| }nt| || }| ddkrtdd| |}n|}t| |}!|!| krnd}"|dur|dkrt d!}#fdd}$t"|$|#||| | |!\} } }}"#||| |k}n3#||||krt t } Wdn 1swY } | |k}d}"| |"7} | d|"7<|rn&|||krnt| ||kr%n| |kr+n ||kr1n||ks|| d<|| d<|| d<|| d <|| d <|| d <|| d <|| d <Wd| S1sawY| S)aPerforms a single optimization step. Args: closure (callable): A closure that reevaluates the model and returns the loss. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print("loss:", loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... >>> opt.step(closure) rrrr NrrrrrrrrrU)dtypeg|=rr0Z strong_wolfez only 'strong_wolfe' is supportedcs|||Sr)r)r4rrrrvr%r&rIszLBFGS.step..obj_func)$r*Zno_gradZ enable_gradrYrZr[r]rOr_r^r( setdefaultrrrGr:getnegZ to_tensorrsubtractmultiplyr6lenpoprrangerrrHr9r2 RuntimeErrorrrSr)%rvrrYrZr[r]rOr_r^r(Z orig_lossrJZ current_evalsrZopt_condrrrrrrrrrr5sZysZnum_oldrqirZbe_irLrPZx_initrIr%rr&stepVs )                      $ &                66z LBFGS.steprcCstd)z}Empty method. LBFGS optimizer does not use this way to minimize ``loss``. Please refer 'Examples' of LBFGS() above for usage.zeLBFGS optimizer does not use this way to minimize loss. Please refer 'Examples' of LBFGS() for usage.)NotImplementedError)rvrJZstartup_programraZ no_grad_setr%r%r&minimize6szLBFGS.minimize) rUrVNrWrDrXNNNNN)rYrrZrr[r\r]rrOrr^rr_r`rarbrcrdrerfrgr`rhri)rhr')rhr)rhr)NNN)rhr)r!r"r#__doc__rrr~rrrrrrr Znon_static_onlyrr __classcell__r%r%rxr&rTes4\ 3 4    arTr)rBrCrDrE)# __future__rr+r- collectionsr functoolsrtypingrrrZtyping_extensionsrr*baser Z optimizerr collections.abcr rZpaddle.nn.cliprZpaddle.regularizerrr__all__rr'r/r6rArSrTr%r%r%r&s:            , q