o 0 i @sxddlZddlmZddlmZddlmZddlmZm Z e e Z er*ddl Z ddZ   d1d eed ed d eed edeffddZ   d1d eed ed d eed edeffddZ   d1d eed ed d eed edeffddZ d2d ed d d eed edeffddZ d2d ed d d eed edeffddZ d2d ed d d eed edeffddZeeeeeedZ  d3dedededeed eef d!d"Zd2d ed eefd#d$Zd2d ed eefd%d&Zd2d ed eefd'd(Zd2d ed eefd)d*Zd2d ed eefd+d,Z d2d ed eefd-d.Z!eeeee e!dZ"d2d ed eefd/d0Z#dS)4Nwraps)Optional)PretrainedConfig)is_torch_availableloggingcs,ddddtfdd}|S)ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. cSst|d}t|jdr|jj}n|jj}||kr8t|ds-|j|j||dd\|_}|jd|jdddS|j ||_ |jd|j dddS) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r original_max_position_embeddings long_inv_freqseq_leninv_freqF persistentN) torchmaxhasattrconfigr max_position_embeddings rope_init_fnr register_bufferoriginal_inv_freqto)self position_idsdevicer r _rl/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_update+s     z6dynamic_rope_update..longrope_frequency_updatecSst|d}||jkr#|j|j||d\}|_|jd|dd||_||jkrD|j|jkrF|j ||_|jd|jdd|j|_dSdSdS)a dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rr r FrN) rrZmax_seq_len_cachedrrZattention_scalingrZoriginal_max_seq_lenrr)rrrr r rrrdynamic_frequency_update>s  z5dynamic_rope_update..dynamic_frequency_updatecsBd|jvr|||jdn |jdkr|||jd|||S)Ndynamic)rlongrope) rope_typer)rxrr r rope_forwardrrwrapperQs   z$dynamic_rope_update..wrapperr)r&r'rr%rdynamic_rope_updates  r(rrz torch.devicer returnz torch.Tensorc Csn|j}t|dd}t|ddp|j|j}t||}d}d|tjd|dtjdj|tj d|}||fS) a  Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). partial_rotary_factor?head_dimNrdtyperr/) rope_thetagetattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrr baser*r,dimattention_factorr rrr _compute_default_rope_parameters\s  ,r<cCs*|jd}t|||\}}||}||fS)a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). factor) rope_scalingr<)rrr r=r r;rrr'_compute_linear_scaling_rope_parameterss r?c Cs|j}t|dd}t|d|j|j}t||}|j}|jd}d} |dur*|}nt|tj r?t |tj ||j |j d}nt||}|||||d||d}d|tjd |dtjd j|tjd |} | | fS) a Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The default sequence length used to update the dynamic RoPE at inference time * rope_scaling (`dict[str, float]`): The standard RoPE scaling parameters, from which `factor` will be accessed. The value of `factor` is used to determine the new base frequency, along with the current sequence length (seq_len), the maximum positional embeddings (max_position_embeddings), and the computed dimensionality (dim) of the rotary embeddings. If seq_len <= max_position_embeddings, this factor has no effect. If seq_len <= max_position_embeddings, this factor effectively stretches the context window using an exponent derived from `dim`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. If `None` or shorter than max_position_embeddings, this value will be overridden by max_position_embeddings. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). r*r+r,r=Nr/rrr-rr.r0)r1r2r3r4r5rr> isinstancerZTensormaximumtensorr/rrr6r7rr8) rrr r9r*r,r:rr=r;r rrr_compute_dynamic_ntk_parameterss$*     $,rDcs|j}t|dd}t|d|j|j}t||}|jd}|jd}|jd} |jd} |jdp8|j} dd d } |d urW| rS| rSt| || | || }n| |}|jd p^d} |jdpfd }ddfdd}dd}|t d|dj |t jd|}d|}d||}|jdd}|| |||| |\}}d ||||dj |t jd}|d |||}||fS)ak Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_scaling (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as avaialble. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunciton with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, the function falls back to `max_position_embeddings`. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r*r+r,r=r;mscalemscale_all_dimr rcSs"|dkrdSd|t|dS)Nrr+g?)mathlog)scalerErrr get_mscale:sz,_compute_yarn_parameters..get_mscaleN beta_fast beta_slowcSs*|t||dtjdt|S)zPInverse dimension formula to find the dimension based on the number of rotationsr-)rGrHpi)Z num_rotationsr:r9rrrrfind_correction_dimLs*z5_compute_yarn_parameters..find_correction_dimcsL||||}||||}|rt|}t|}t|dt||dfS)z.Find dimension range bounds based on rotationsrr)rGfloorceilrmin)Zlow_rotZhigh_rotr:r9rtruncatelowhighrOrrfind_correction_rangePs   z7_compute_yarn_parameters..find_correction_rangecSs>||kr|d7}tj|tjd|||}t|dd}|S)NgMbP?r.rr)rr6float32clamp)rRrr:Z linear_funcZ ramp_funcrrrlinear_ramp_factorYs z4_compute_yarn_parameters..linear_ramp_factorrr-r0rST)r) r1r2r3r4r5r>getrr8rr6r)rrr r9r*r,r:r=r;rErFr rJrKrMrWrZZ pos_freqsZinv_freq_extrapolationZinv_freq_interpolationrSrTrUZinv_freq_extrapolation_factorr rrVr_compute_yarn_parameterss>8         "    r\cCs|j}t|dd}t|d|j|j}t||}|jd}|jd}|jd} |jd} t|dd } r=|j| } n|j} | d urZ| dkrKd} nt d t | t | } |rj|| krjt j |t j |d } n t j |t j |d } t jd |d t j|d |} d| || }|| fS)a Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_scaling (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r*r+r, long_factor short_factorr=r;r Nrr@rr-)r1r2r3r4r5r>r[rrGsqrtrHrrCrXr6r7r8)rrr r9r*r,r:r]r^r=r;r Z ext_factorsZinv_freq_shaper rrr_compute_longrope_parametersss*/        r`cCst|||\}}|jd}|jd}|jd}|jd}||} ||} dtj|} t| | k|||} || |||} d| | || | }| | k| | k}t||| } | |fS)ap Computes the inverse frequencies for llama 3.1. Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * rope_scaling (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `factor` (`float`, *optional*): The scaling factor applied to the inverse frequencies when 1) the wavelength is greater than `low_freq_wavelen` prior to smoothing, and 2) to all inverse frequencies during smoothing. * `high_freq_factor` (`float`): The scale factor used to compute `high_freq_wavelen` and the value for the denominator of the smoothing factor prior to the `low_freq_factor` shift. * `low_freq_factor` (`float`): The scale factor used to compute `low_freq_wavelen` and the shift applied to the numerator and denominator of the smoothing factor. frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. If not provided, the function falls back to `max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r=low_freq_factorhigh_freq_factorr r-r)r<r>rGrNrwhere)rrr r r;r=rarbZold_context_lenZlow_freq_wavelenZhigh_freq_wavelenZwavelenZinv_freq_llamaZ smooth_factorZsmoothed_inv_freqZis_medium_freqrrr_compute_llama3_parameterss*    rd)defaultZlinearr!yarnr"Zllama3r# received_keys required_keys optional_keys ignore_keyscCsd|vr|dh8}|d|dur||8}||}|r&td|d||dur1|||}n||}|rDtd|d|dSdS)zYCompare the received keys in `config.rope_scaling` against the expected and optional keystyper#Nz9Missing required keys in `rope_scaling` for 'rope_type'='z': z5Unrecognized keys in `rope_scaling` for 'rope_type'=')addKeyErrorloggerwarning)r#rgrhrirjZ missing_keysZ unused_keysrrr_check_received_keyss  rpcCs@|j}|d|dd}dh}t|}t||||ddS)Nr#rkrj)r>r[setkeysrp)rrjr>r#rhrgrrr!_validate_default_rope_parameters0s  rtcCsx|j}|d|dd}ddh}t|}t||||d|d}|dus0t|tr0|dkr:td|dSdS)Nr#rkr=rqr+8`rope_scaling`'s factor field must be a float >= 1, got r>r[rrrsrprAr8rnro)rrjr>r#rhrgr=rrr(_validate_linear_scaling_rope_parameters8s rwcCs|j}|d|dd}ddh}dh}t|}t|||||d|d}|dus4t|tr4|dkr>td|dSdS)Nr#rkr=r rqr+rurv)rrjr>r#rhrirgr=rrr)_validate_dynamic_scaling_rope_parametersDs rxc Cs|j}|d|dd}ddh}hd}t|}t|||||d|d}|dus5t|tr5|dkr=td||d}|durWt|trO|d krWtd ||d } | durmt| tsmtd | |d } | durt| tstd| | pd| pdkrtd| d| d|jd} | dur|j | } | |krt d|d| d|ddSdSt ddS)Nr#rkr=>rSrKrMr;rEr rFrqr+rur;rL`rope_scaling`'s attention_factor field must be a float greater than 0, got rKz6`rope_scaling`'s beta_fast field must be a float, got rMz6`rope_scaling`'s beta_slow field must be a float, got rLrzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r zHThe explicitly set RoPE scaling factor (config.rope_scaling['factor'] = z) does not match the ratio implicitly set by other parameters (implicit factor = post-yarn context length / pre-yarn context length = config.max_position_embeddings / config.rope_scaling['original_max_position_embeddings'] = z). Using the explicit factor (z) in YaRN. This may cause unexpected behaviour in model usage, please correct the 'max_position_embeddings' fields in the model config.a~config.rope_scaling['original_max_position_embeddings'], the pre-yarn context length, is unset. We will **assume** config.max_position_embeddings holds the pre-yarn context length. Some use cases may expect config.max_position_embeddings to hold the post-yarn context length (pre-yarn context length * factor) -- we recommend updating both fields for optimal downstream model usage.) r>r[rrrsrprAr8rnror warning_once) rrjr>r#rhrirgr=r;rKrMr Zimplicit_factorrrr_validate_yarn_parametersRsR       r{cCs|j}|d|dd}hd}hd}t|}t|||||dt|dd}t|d|j|j}t||} |d } t | t sUt d d | DrUt d | t| | d krlt d| d dt| |d} t | t st dd | Drt d| t| | d krt d| d dt| t|drt ddS|d} | durt dnt | tr| dkrt d| |d} | durt | tr| dkrt d| dSdSdS)Nr#rk>r]r#r^>r=r;r rqr*r+r,r^cs|] }t|ttfVqdSNrAr5r8.0r$rrr z0_validate_longrope_parameters..zC`rope_scaling`'s short_factor field must be a list of numbers, got r-z5`rope_scaling`'s short_factor field must have length z, got r]csr|r}r~rrrrrrzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.r=z1Missing required keys in `rope_scaling`: 'factor'rur;gry)r>r[rrrsrpr2r3r4r5rAlistallrnrolenrrzr8)rrjr>r#rhrirgr*r,r:r^r]r=r;rrr_validate_longrope_parameterssH         rc Cs6|j}|d|dd}hd}t|}t||||d|d}|dus0t|tr0|dkr8td||d}|d }|dusIt|tsQtd ||dusZt|tsbtd |||krqtd |d ||d} | dus~t| t std| | |j krtd| d|j dSdS)Nr#rk>rbrar r#r=rqr=r+rurarbz<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) r>r[rrrsrprAr8rnror5r) rrjr>r#rhrgr=rarbr rrr_validate_llama3_parameterssL  rcCsdt|dd}|dur dS|d|dd}t|}|dur'|||ddStd|ddS) zO Validate the RoPE config arguments, given a `PretrainedConfig` object r>Nr#rkrerqzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')r2r[ROPE_VALIDATION_FUNCTIONSrnro)rrjr>r#Z validation_fnrrrrope_config_validations   r)NNNr})NN)$rG functoolsrtypingrZconfiguration_utilsrutilsrrZ get_logger__name__rnrr(r5tupler8r<r?rDr\r`rdZROPE_INIT_FUNCTIONSstrrrrprtrwrxr{rrrrrrrrs    ?  ,  ,  E  ~  S  E  B2&