o 81 i|@sdddlZddlmZddlZddlmZddlmZmZddl m Z zddl m Z m Z mZmZmZWneyEd\ZZd\Z Z dZYnwz ddlmZmZWn ey]d\ZZYnwzddlmZWn eyqdZYnwd d ZGd d d ejZGd ddejZGdddejZGdddejZddZGdddejZGdddejZdS)N)partial) rearrangerepeat)get_dim_for_local_rank)flash_attn_kvpacked_funcflash_attn_qkvpacked_funcflash_attn_varlen_kvpacked_func flash_attn_varlen_qkvpacked_funcflash_attn_with_kvcacheNN)ColumnParallelLinearRowParallelLinear)RotaryEmbeddingcCs\dd}t|r||Sdtt|}||td|dddd||S)Ncs6ddt|d  fddt|DS)Ncsg|]}|qSr).0iratiostartrb/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/flash_attn/modules/mha.py )szCget_alibi_slopes..get_slopes_power_of_2..)mathlog2range)nheadsrrrget_slopes_power_of_2&sz/get_alibi_slopes..get_slopes_power_of_2rr)rr is_integerfloorget_alibi_slopes)rrZclosest_power_of_2rrrr %s r cs8eZdZdZ      d fdd Zd dd ZZS) FlashSelfAttention|Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax attention. (default: 1/sqrt(d_keys) where d_keys is computed at runtime) attention_dropout: The dropout rate to apply to the attention (default: 0.0) FNr%csbttdus JdtdusJd||_||_t||_|j d|dd||_ ||_ dSNzFlashAttention is not installed alibi_slopesF) persistent) super__init__r rcausal softmax_scalennDropoutdropregister_buffer window_size deterministic)selfr+r,attention_dropoutr1r'r2 __class__rrr*@   zFlashSelfAttention.__init__c Cs|jtjtjfvs J|jsJ|dur|jn|}|du}|jdur*|jtj|_|rY|jtj ks4J|dus:Jt |t sAJt ||||j rL|jjnd|j||j|j|jd St||j rb|jjnd|j||j|j|jdS)aoImplements the multihead softmax attention. Arguments --------- qkv: The tensor containing the query, key, and value. If cu_seqlens is None and max_seqlen is None, then qkv has shape (B, S, 3, H, D). If cu_seqlens is not None and max_seqlen is not None, then qkv has shape (total, 3, H, D), where total is the sum of the sequence lengths in the batch. causal: if passed, will override self.causal cu_seqlens: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into qkv. max_seqlen: int. Maximum sequence length in the batch. Returns: -------- out: (total, H, D) if cu_seqlens is not None and max_seqlen is not None, else (B, S, H, D). Nr#r,r+r'r1r2)dtypetorchfloat16bfloat16is_cudar+r'tofloat32int32 isinstanceintr trainingr/pr,r1r2r)r3qkvr+ cu_seqlens max_seqlenunpaddedrrrforwardSs<    zFlashSelfAttention.forward)FNr#r$NF)NNN__name__ __module__ __qualname____doc__r*rI __classcell__rrr5rr!5s r!csBeZdZdZ      d fdd Z     d dd ZZS) FlashCrossAttentionr"FNr#r$csbttdus JdtdusJd||_||_t||_|j d|dd||_ ||_ dSr&) r)r*rrr+r,r-r.r/r0r1r2)r3r+r,r4r'r1r2r5rrr*r7zFlashCrossAttention.__init__c Csh|jtjtjfvs J|jr|jsJ|dur|jn|}|du}|jdur-|jtj|_|rz|jtj ks7J|dus=Jt |t sDJ|dusJJ|jtj ksRJ|dusXJt |t s_Jt |||||||j rm|jjnd|j||j|j|jd S|jd|jd} } |jd} |jd| kr|jd|jdksJt|||j r|jjnd||j|j|j|jdS) aImplements the multihead softmax attention. Arguments --------- q: The tensor containing the query. (B, Sq, H, D) kv: The tensor containing the key and value. (B, Sk, 2, H_k, D) causal: if passed, will override self.causal cu_seqlens: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. max_seqlen: int. Maximum sequence length in the batch of q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_k: int. Maximum sequence length in the batch of k and v. Nr#r8rr)r+r,r'r1r2)r9r:r;r<r=r+r'r>r?r@rArBrrCr/rDr,r1r2shaper) r3qkvr+rFrGZ cu_seqlens_kZ max_seqlen_krH batch_sizeseqlen_qseqlen_krrrrIsR     &zFlashCrossAttention.forward)FNr#Nr$F)NNNNNrJrrr5rrPs rPc,eZdZdZd fdd Zd ddZZS) SelfAttentionr"FNr#c&t||_||_t||_dSNr)r*r+r,r-r.r/r3r+r,r4r5rrr* zSelfAttention.__init__cCs|jd|jd}}|dur|jn|}|jdd\}}}|jp)dt|jd} td||| } |durQtj||fd | j | j d } | |d | t | d } |rkt tj||fd | j d d} | | j| j d} tj| d|j d} || }td||}|S)auImplements the multihead softmax attention. Arguments --------- qkv: The tensor containing the query, key, and value. (B, S, 3, H, D) causal: if passed, will override self.causal key_padding_mask: boolean mask to apply to the attention weights. True means to keep, False means to mask out. (B, S) rrQNrdim?r%bthd,bshd->bhtsr9devicer#b s -> b 1 1 srf)r9rar9bhts,bshd->bthd)rSr+unbindr,rsqrtr:einsumfullr9rf masked_fill_rZtriur>softmaxr/)r3rEr+key_padding_maskrVseqlenrTkvr,scores padding_mask causal_mask attentionattention_dropoutputrrrrIs(   zSelfAttention.forwardFNr#r rJrrr5rrZ rZcrY) CrossAttentionr"FNr#cr[r\r]r^r5rrr*%r_zCrossAttention.__init__cCs|jd|jd}}|dur|jn|}|jd}|jd|kr*|jd|jdks,J|jd|jdkrEt|d|jd|jdd}|jdd \}} |jpYd t|jd } td ||| } |durtj ||fd | j | j d} | |d| t | d} |rt tj||j tjdd} tj||j tjd}|dur|nt |d d}|| ||k}| |d } tj| d | j d}||}td|| }|S)aImplements the multihead softmax attention. Arguments --------- q: The tensor containing the query. (B, Sq, H, D) kv: The tensor containing the key and value. (B, Sk, 2, H_k, D) causal: if passed, will override self.causal key_padding_mask: boolean mask to apply to the attention weights. True means to keep, False means to mask out. (B, Sk) rrQNrRrrz... hkv d -> ... (hkv g) d)gr`rbr%rcrdrer#rgrfr9zs -> s 1z b -> b 1 1 1rirj)rSr+rrkr,rrlr:rmrnr9rfrorZarangelongsumZ masked_fillrpr/)r3rTrUr+rqrVrWrXrsrtr,rurvZrow_idxZcol_idxskrwrxryrzrrrrI+s<  &   zCrossAttention.forwardr{r rJrrr5rr}r|r}c Cs|jdd\}}||jvr$tj|j|jd|||j|jd}||j|<n|j|}|j}||jd}|j }||jd} ||jdksFJ| |jdksOJ|dusUJ|||||| df<|||d| dfS)Ukv: (batch_size, seqlen, 2, nheads, head_dim) or (batch_size, 1, 2, nheads, head_dim)NrrerrQ.) rSkey_value_memory_dictr:emptyZmax_batch_sizerGr9rfZbatch_size_offset seqlen_offset) rUinference_params layer_idx num_headshead_dimkv_cacheZ batch_startZ batch_endZsequence_startZ sequence_endrrr_update_kv_cacheXs,   rcseZdZdZ                     d dfd d Zdd d ZddZddZddZ      dddZ Z S)MHA-Multi-head self-attention and cross-attentionNFTr#r@r$returncsR||d}t||_||_| |_| |_| |_| |_||_||_ ||_ |r7|s-Jdt j t ||d}nd}|dkrC|sCJd||_|durL|n||_|j|jdks[Jd|j|dksfJd |j||_|j|jd |j}d |j|j}|jdkr|rJd tdusJd t|j| |||d |_|rtt||dnt}|rtt||dnt}|jstj||fd|i||_ntj||fd|i||_tj||fd|i||_|jr |j|jkrtj||dd |d|_ntj||dd |d|_tj||dd |d|_ || ||d|_!|| ||d|_"tj||fd|i||_#dS)aX num_heads_kv: can be used to toggle MQA / GQA. If None, use num_heads. return_residual: whether to return the input x along with the output. This is for performance reason: for post-norm architecture, returning the input allows us to fuse the backward of nn.Linear with the residual connection. r#ALiBi code path requires flash_attnrhNr$>Local (sliding window) attention code path requires flash_attnr+num_heads must be divisible by num_heads_kv(embed_dim must be divisible by num_headsrz>MHA with rotary embedding does not support cross-attention yetrotary_emb is not installedbaseZ scale_base interleavedrfr'r1biasr)Z kernel_sizepaddinggroupsr+r,r4)$r)r* embed_dim cross_attnr+rdwconvrotary_emb_dimuse_flash_attnreturn_residual checkpointingr:tensorr r num_heads_kvrr rotary_embrr!rZrPr}r-ZLinearWqkvWqWkvZConv1d dwconv_qkvdwconv_q dwconv_kv inner_attninner_cross_attnout_proj)r3rrrr qkv_proj_bias out_proj_biasdropoutr,r+rrrrotary_emb_baserotary_emb_scale_baserotary_emb_interleaved use_alibir1Z fused_bias_fcrrrrfr9factory_kwargsr'qkv_dimZkv_diminner_attn_clsinner_cross_attn_clsr5rrr*xs           z MHA.__init__c C<|dur |jjjn|}|jjj}tj||d|j|j||dSNrre)rweightr9rfr:rrrr3rVrGr9rfrrrallocate_inference_cache zMHA.allocate_inference_cachecCs.|jrJd|jdusJdt|||jS)rz&Generation does not support dwconv yetN0Generation requires layer_idx in the constructor)rrrr3rUrrrrrszMHA._update_kv_cachec CB|dur |jdks J|jsJ|jdkr5|jjdusJd|jj|j|j|jd|jj |jj }}nd\}}|j d}|j |j d|}|jdurT|jd|n|j}t|jdd} t||dddddf|dddddf|dddddf|dddddf||||jj|jj|jdkr|jjnd| d } | S z Fast path that combine 3 steps: apply rotary to Q and K, update kv cache, and apply attention. q: (batch_size, seqlen_q, nheads, head_dim) kv: (batch_size, seqlen_k, 2, nheads_kv, head_dim) Nrz$This code path does not support xPosrr r'rQF) rotary_cos rotary_sin cache_seqlensr,r+Zrotary_interleavedr'rrrrscaleZ_update_cos_sin_cacherGrfr9Z _cos_cachedZ _sin_cachedrSrrlengths_per_samplegetattrrr r,r+r r3rTrUrrrbatchrrr'contextrrr&_apply_rotary_update_kvcache_attention>     z*MHA._apply_rotary_update_kvcache_attentionc Cs|jdks tdus |js|||}|||S|jd}|j|jd|}|jdur3|jd|n|j}t |jdd}t||dddddf|dddddf|dddddf|dddddf||jj |jj |d Sz/Write kv to inference_params, then do attentionrNr'rQ)rr,r+r') rr rrrrSrrrrr,r+)r3rTrUrrrrr'rrr_update_kvcache_attentions0     zMHA._update_kvcache_attentioncKsd|dur!|dus J|dusJ|jsJ|jrJ|jdks!J|dur6|dus+J|dus1J|jr6J|durO|dus@J|durH|dusJJ|jrOJ|jrY||d|nd|i|}|duredn |jdurm|jn|j} |durw|jnd} |jdd\} } |js<|j|j kr<|dur|dusJ| |} |jrt | t | ddddfd  } t | d d |jd } |dus|jdks|jdks|jd dks|js |jdkr|j| | | d} |dur|js|j| fi|}n,tjjj|j| fi|}n|| dddddf| ddddddf|}n|| dddddf| ddddddf|}n|jr_||durI|n|dd|f}||dur[|n|}n&|j|j kshJ| |} | dd|j |jf}| d|j |jdf}t |d|jd}t |dd|jd}|jrt |t |ddddfd  }t |t |ddddfd  }|dus|jdks|jdks|jd dks|js|jdkr|j||| | d\}}|dur|js|j||fi|}ntjjj|j||fi|}n||||}n||||}|t |d}|js.|S||fS)a Arguments: x: (batch, seqlen, hidden_dim) (where hidden_dim = num heads * head dim) if cu_seqlens is None and max_seqlen is None, else (total, hidden_dim) where total is the is the sum of the sequence lengths in the batch. x_kv: (batch, seqlen, hidden_dim), only applicable for cross-attention. If None, use x. cu_seqlens: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into x. Only applicable when using FlashAttention. max_seqlen: int. Maximum sequence length in the batch. key_padding_mask: boolean mask, True means to keep, False means to mask out. (batch, seqlen). Only applicable when not using FlashAttention. mixer_subset: for cross-attention only. If not None, will take a subset of x before applying the query projection. Useful for e.g., ViT where we only care about the CLS token in the last layer. inference_params: for generation. Adapted from Megatron-LM (and Apex) https://github.com/NVIDIA/apex/blob/3ff1a10f72ec07067c4e44759442329804ac5162/apex/transformer/testing/standalone_transformer_lm.py#L470 Nr)rFrGrqrzb s d -> b d s.rzb d s -> b s dz ... (three h d) -> ... three h drthreedrrGrQ... (h d) -> ... h dr ... (two hkv d) -> ... two hkv dtworz... h d -> ... (h d))rrrrrrGrSrrrrrr contiguousrrrrr:utils checkpointrrrrrrrrr)r3xZx_kvrqrFrGZ mixer_subsetrkwargsrrotary_max_seqlenrrrrErrTrUoutrrrrI=s                ..$     z MHA.forward)NFTTr#NFNFrrNFFr$FFFFNNrNr\)NNNNNN rKrLrMrNr*rrrrrIrOrrr5rrusJ k ("rcsveZdZdZ                  d dfd d Zdd d ZddZddZddZdddZ Z S) ParallelMHArNTr#Frrr$rcs6||d}t||_| |_| |_| |_||_||_||_| |_ t j ||_||_|j|jdks:Jd|dur@|n||_|j|jdksOJdt|j|j |j|_t|j|j |j|_|j||_|j|jd|j}|r|s|Jdt|j|j }t jt||j||jd||d}nd}|d kr|sJd |jdkrtdusJd t|j| | ||d |_tdustdurtd t|||f|||j|j|jdd||_|rtt ||dnt!}|rtt"||dnt#}|| ||d|_$|| ||d|_%t|||f|||jd||_&dS)NrrrrrrrQrhr$rrrzfused_dense is not installed)rsequence_parallelZ multiple_ofrr)'r)r*rr+rrrr process_groupsizeZ world_sizer: distributedZget_rankZ local_rankrrrnum_heads_per_ranknum_heads_kv_per_rankrrceilrr rrr r ImportErrorrrr!rZrPr}rrr)r3rrrrrrrr,r+rrrrrrr1rrrrfr9rrZnum_heads_localr'rrr5rrr*s           zParallelMHA.__init__c Crr)rrr9rfr:rrrrrrrr8rz$ParallelMHA.allocate_inference_cachecCs |jdus Jdt|||jS)rNr)rrrrrrrEszParallelMHA._update_kv_cachec CrrrrrrrrJrz2ParallelMHA._apply_rotary_update_kvcache_attentionc Cs|jdks|js|||}|||S|jd}|j|jd|}|jdur/|jd|n|j}t|jdd}t ||dddddf|dddddf|dddddf|dddddf||jj |jj |d }|Sr) rrrrrSrrrrr r,r+) r3rTrUrrrrr'rrrrrrs,     z%ParallelMHA._update_kvcache_attentionc Ks||}|durt|d|d}|durdn |jdur|jn|j}|dur(|jnd}|j|jkrt|dd|jd}|dusQ|jdksQ|jdksQ|jddksQ|j s|jdkr^|j |||d }|dur}|j so|j |fi|}nt jjj|j |fi|}n||dddddf|ddddd df|}n||dddddf|ddddd df|}nt|d d|j|jfd |jd } t|d |j|jdfdd|jd} |dus|jdks|jdks|jddks|j s/|jdkr|j | | ||d \} } |dur'|j s|j| | fi|}nt jjj|j| | fi|}n|| | |}n|| | |}t|d}|durEt|d}||} | S)ae Arguments: x: (batch, seqlen, hidden_dim) (where hidden_dim = num heads * head dim) if seqlen=None. If seqlen is not None, x is (batch * seqlen, hidden_dim). This is so that when we split x during sequence parallel, we split the batch * seqlen dimension (in case batch is small). Nz(b s) ... -> b s ...)srz b s (three h d) -> b s three h drrrrrQ.rrrrrzb s h d -> b s (h d)zb s d -> (b s) d)rrrrrGrrrrrrrrr:rrrrrrr) r3rrrrrrErrrrTrUrrrrrIs     ..       zParallelMHA.forward)NTTr#NFNrrNFFr$FFTNNrr\r rrrr5rrs8 r (r) r functoolsrr:Ztorch.nnr-ZeinopsrrZflash_attn.utils.distributedrZ flash_attnrrrr r rZflash_attn.ops.fused_denser r Zflash_attn.layers.rotaryrr Moduler!rPrZr}rrrrrrrsB        Pa4>P