o )iD/%@sddlmZmZddlZddlmZddlmZdddd d dd dddd d d dejdejdejdejdejdede de de de de de de deejdeejdeejde d e f$d!d"Z dS)#)OptionalTupleN)+BlockDiagonalCausalWithOffsetPaddedKeysMask)_is_triton_availableg@g?Fg@g0@g@@T) theta linear_scaleuse_dynamic_scalingdynamic_old_context_lendynamic_scale_factordynamic_low_freq_factordynamic_high_freq_factorout_q first_seqposseqpos adjacentsinternal_dtypexqxkxvcache_kcache_v attn_biasrr r r r r rrrrrrc 4 Cstr|js|js|js|js|js| durtdts Jddl}ddlm}|jj d}|j j d}|j }|dvrAtd| }| }| }| }| }|j }|j }|d }|}|d dkrid}|} |dkry|d dkry|d } |d kr|j \}!}"}#}$|"|ksJ|d|||$fkrtd |d d|||$f|j d|||$fkrtd |j d d|||$f|d|| |$fkrtd|j d|| |$fkrtdd}%d|#|$|$df}&n|j \}!}"}%}#}$|"|ksJ|d||%||$fkrtd |d d||%||$f|j d||%||$fkrtd |j d d||%||$f|d||%| |$fkr7td|d d||%| |$f|j d||%| |$fkrRtd|j d d||%| |$fd|#|$|%|#|$|$df}&|!dkrhtd|ddkrstd|ddkr~td|ddkrtd|ddkrtd|ddkrtd|#d|}'|'|}(|#})| dur||j } n| j |j krtd| }&|&ddkrtd| dusJt|jj d}*|dur|durtdd}+|dur|j |*fkr t|j },td|,d|*d| d}+n!|dur3|j |fkr.t|j },td|,d|d| d}+d |}-t|-||$}.t|.d!}.t|.d"}.tt|.d#dd$}/|j}0|jj}1|j j}2|j j}3|1j|0ksv|2j|0ksv|3j|0krztd%|d&vsJtj|j||jj|*|'|%fg|||| |||1|2|3||||r| nd|r| nd|r| nd|r| nd|||)|(|%|$|d|d'kr|dnd|d |d|d'kr|dnd|d |d|d'kr|dnd|d |d|d'kr|dnd|d |d|d'kr-|dnd|d |1 d|2 d|3 d|&d|d'krO|&dnd|&d |+|Rd(dd|.||/d)Wd| S1spwY| S)*u Performs RoPE (rotary embeddings) and kv-cache emplacement for a heterogeneous batch for inference in the style given by BlockDiagonalCausalWithOffsetPaddedKeysMask. The batch is concatenated along the sequence dimension, so the actual dim-0 length of all tensors is 1. xq, xk and xv should be (1, slen, n_heads, dim), where xq's n_heads can differ from xk and xv. This function places the roped xk in the right place in cache_k, and xv (unmodified) in the right place in cache_v, and returns out_q (the roped xq) such that things are ready to call xformers.ops.memory_efficient_attention( out_q, cache_k, cache_v, attn_bias=attn_bias ) This functionality is experimental. Its API might be changed without warnings. Use it at your own risk. Arguments: xq: tensor of queries to apply rope to xk: tensor of keys to apply rope to xv: tensor of values to copy into cache_v cache_k: cache of keys, MODIFIED IN PLACE cache_v: cache of values, MODIFIED IN PLACE attn_bias: details the layout of caches. Used to determine frequencies for the RoPE calculation as well as the locations in cache_k and cache_v to write to. Must be on the device. first_seqpos: Optionally a tensor containing the sequence position of the beginning of the cache for each batch element. Providing a tensor of zeros is the same as providing None. This affects the numerical calculation but not which memory locations are read or written. seqpos: Optionally a 1D tensor containing the sequence position of each query. This should have length equal to xq.shape[1] . This affects the numerical calculation but not which memory locations are read or written. adjacents: If True, the inputs are in adjacent pairs along the final dim axis. This is like the released LLaMA model. If False, the dim axis is split in two equal pieces. I.e. the features are ordered with all the real parts before all the imaginary parts. This matches HuggingFace, e.g. https://github.com/huggingface/transformers/blob/ f143037789288ba532dada934a118e648e715738/ src/transformers/models/llama/modeling_llama.py#L126-L130 linear_scale: A scaling factor to apply to the sequence ids when computing the RoPE frequencies. When set to K, all sequence indices are divided by K. use_dynamic_scaling: If true, dynamic scaling in use, using a scaling like “YaRN: Efficient Context Window Extension of Large Language Models” dynamic_old_context_len: used with use_dynamic_scaling dynamic_scale_factor: used with use_dynamic_scaling dynamic_low_freq_factor: used with use_dynamic_scaling dynamic_high_freq_factor: used with use_dynamic_scaling internal_dtype: set to "f32" or "f64" to enforce dtype in the calculation NzGradients not supported.r)_rope_padded_kernel)zUnexpected xq dimensionrzunexpected k shape z : expected zunexpected v shape zunexpected cache_k shapezunexpected cache_v shapezunexpected cache_k shape zunexpected cache_v shape zHExpected batch size dimension to be 1 as batches should be concatenated.zEach q head must be contiguouszEach k head must be contiguouszEach v head must be contiguousz$Each cache_k head must be contiguousz$Each cache_v head must be contiguousrzUnexpected shape of out_qz"Each out_q head must be contiguousz0seqpos and first_seqpos may not both be providedzfirst_seqpos.shape z but (z ,) expected.z seqpos.shape iiz:`attn_bias` must be on the same device as the other inputs)rZf32Zf64rF)Zconst_batch_stridesZcache_padding_lengthZ seqlenk_shift BLOCK_SIZEr num_warps)torchZis_grad_enabledZ requires_grad ValueErrorrtritonZ_triton.rope_padded_kernelsrZ q_seqinfoZ seqstart_pyZ k_seqinfondimZstrideshapeZ new_emptylentupleZ element_sizeminZnext_power_of_2maxdeviceZseqstartZseqlencudaZ max_seqlen)4rrrrrrrr r r r r rrrrrrr'rZn_total_queriesZ cache_lengthr(Z xq_strideZ xk_strideZ xv_strideZcache_k_strideZcache_v_strideZ cache_k_shapeZxk_shapeZ n_kv_headsZexpected_kv_headsZexpected_cache_headsZbszZq_lenZ n_q_headsdimZn_groupsZ out_q_strideZ n_total_headsZv_startZk_startZ logical_bszZ stride_seqposr)ZMAX_FUSED_SIZEr#r$r.Z seqstartqZ seqstartkZseqlenkr1d/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/xformers/ops/rope_padded.py rope_paddedsP                                    !"#$%&'()*+,-./0 88r3) typingrrr%Zxformers.ops.fmha.attn_biasrrrZTensorfloatboolstrr3r1r1r1r2sj