o 81 i2@sBddlmZmZmZmZddlZddlmZddlZe dddkZ e r+ddl m Z nddlZ dd Zd d Zd d ZejdkrIejjZejjZndkdddddZdkdddddZeZeZedddddejdejdejdedededed ed!ed"eejd#ed$eejejejejffd%d&Zeddejdejdejdedededed ed!ed"eejd#ed$eejejejejffd'd(Zejdkrej j!jZ"neZ"ed)ddd * * +  ,    ,dldejdejdejd-ejd.ejd/ed0ededededed ed!ed"eejd#ed1eejd2eejd3eejd4ed$eejejejejff(d5d6Z#ed) * * +  ,    ,dldejdejdejd-ejd.ejd/ed0ededededed ed!ed"eejd#ed1eejd2eejd3eejd4ed$eejejejejff(d7d8Z$ejdkrej j!j#Z%ne#Z%ed9d:dd dkd;ejdejdejdejdeejd?eejd@eejdedededed ed!ed"eejdAedBeejd$ejf&dCdDZ&ed9 dkd;ejdejdejdejdeejd?eejd@eejdedededed ed!ed"eejdAedBeejd$ejf&dEdFZ'ejdkr+ej j!j&Z(ne&Z(edGd:dd  ,dmd;ejdejdejdejdeejd?eejd@eejd-ejd.ejd/ed0ededededed ed!ed"eejdAedBeejd4ed$ejf0dHdIZ)edG  ,dmd;ejdejdejdejdeejd?eejd@eejd-ejd.ejd/ed0ededededed ed!ed"eejdAedBeejd4ed$ejf0dJdKZ*ejdkrej j!j)Z+ne)Z+GdLdMdMej,j-Z.GdNdOdOej,j-Z/GdPdQdQej,j-Z0GdRdSdSej,j-Z1GdTdUdUej,j-Z2GdVdWdWej,j-Z3 +  , X +  , ,dndYdZZ4 +  , X +  , ,dnd[d\Z5 +  , X +  , ,dnd]d^Z6 +  , X +  , ,dnd_d`Z7 +  , X +  , ,dndadbZ8 +  , X +  , , dodcddZ9          , X + e  ,dpdfeeeejfdgeejdheejd1eejfdidjZ:dS)q)OptionalSequenceTupleUnionNZ!FLASH_ATTENTION_TRITON_AMD_ENABLEFALSETRUE) interface_facCs"|dur|ddkr|S|S)Nr)stride contiguous)xrk/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/flash_attn/flash_attn_interface.pymaybe_contiguouss"rc Cs|dksJtj|\}}|dko|dk}|dko|dk}|dko%|dk}|dkr,dS|dkr6|s4dSdS|dkr.wrapr)namefn mutates_argsr"r#r(rrrnoop_custom_op_wrapper<r-)lib _stacklevelcCsdd}|dur |S|S)NcSr$r%rr&rrrr(Cr)z(noop_register_fake_wrapper..wrapr)opr+r/r0r(rrrnoop_register_fake_wrapperBr.r2zflash_attn::_flash_attn_forwardrr)r,r"qkv dropout_p softmax_scalecausalwindow_size_leftwindow_size_rightsoftcap alibi_slopesreturn_softmaxreturnc CsPdd|||fD\}}}t|||d| ||||||| d \} } } }| | | |fS)NcSg|]}t|qSrr.0r rrr Zz'_flash_attn_forward..)flash_attn_gpuZfwd)r3r4r5r6r7r8r9r:r;r<r=out softmax_lseS_dmask rng_staterrr_flash_attn_forwardLs" rJc Csdd|||fD\}}}|j\} } } }|jd}t|}tj| | | ftj|j|jd}tjd|j|j|jd}| rQtj| | t| dt|df|j|j|jd}tjdtj |jd}||||fS) NcSr?rr@rArrrrC{rDz,_flash_attn_forward_fake..rdtyperlayoutrrrLr) shaper empty_likeemptyfloat32rrMrLr!int64)r3r4r5r6r7r8r9r:r;r<r= batch_sizeseqlen_q num_headsZ head_sizeZseqlen_krFrGprIrrr_flash_attn_forward_fakems  , r[z&flash_attn::_flash_attn_varlen_forwardr F cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_k block_table leftpad_k seqused_k zero_tensorscCs`dd|||fD\}}}t|||d|||||| |||||| | | | |d\}}}}||||fS)NcSr?rr@rArrrrCrDz._flash_attn_varlen_forward..)rEZ varlen_fwd)r3r4r5r]r^r_r`r6r7r8r9r:r;r<r=rarbrcrdrFrGrHrIrrr_flash_attn_varlen_forwards2 recCsdd|||fD\}}}|du}|d}|j\}}}t|}tj||ftj|j|jd}tjd|j|j|jd}t |d}t |d}|rXtj||||f|j|j|jd}tjdtj |jd}||||fS) NcSr?rr@rArrrrCrDz3_flash_attn_varlen_forward_fake..rrKrNrrOrQ) numelrRrrSrTrUrrMrLr!rV)r3r4r5r]r^r_r`r6r7r8r9r:r;r<r=rarbrcrdZpaged_kvrWtotal_qrY_rFrGrZZseqlen_q_roundedZseqlen_k_roundedrIrrr_flash_attn_varlen_forward_fakes       riz flash_attn::_flash_attn_backward)dqdkdvdoutrFrGrjrkrl deterministicrIcCs\dd|||||fD\}}}}}t||||||||||| | | | | ||d|\}}}}|S)NcSr?rr@rArrrrCrDz(_flash_attn_backward..)rEZbwd)rmr3r4r5rFrGrjrkrlr6r7r8r9r:r;r<rnrI softmax_drrr_flash_attn_backwards8"rpcCsdd|||||fD\}}}}}|durt|}|dur#t|}|dur,t|}|j\}}}}tj||t|df|jtjd}|S)NcSr?rr@rArrrrC6rDz-_flash_attn_backward_fake..rrrL)rrSrRrTr!rrU)rmr3r4r5rFrGrjrkrlr6r7r8r9r:r;r<rnrIrWrXrYrhrorrr_flash_attn_backward_fake!s"    rrz'flash_attn::_flash_attn_varlen_backwardcCsfdd|||||fD\}}}}}t|||||||||| | || | | |||||||d|\}}}}|S)NcSr?rr@rArrrrCdrDz/_flash_attn_varlen_backward..)rEZ varlen_bwd)rmr3r4r5rFrGrjrkrlr]r^r_r`r6r7r8r9r:r;r<rnrIrdrorrr_flash_attn_varlen_backwardIsB"rscCsdd|||||fD\}}}}}| d}|j\}}}|dur&t|}|dur/t|}|dur8t|}tj||d|f|jtjd}|S)NcSr?rr@rArrrrCrDz4_flash_attn_varlen_backward_fake..rrrq)rfrRrrSrTrrU)rmr3r4r5rFrGrjrkrlr]r^r_r`r6r7r8r9r:r;r<rnrIrdrWrgrYrhrorrr _flash_attn_varlen_backward_fakes"      rtc@$eZdZeddZeddZdS)FlashAttnQKVPackedFuncc  Csv| o|j} |dur|jdd}|dddddf|dddddf|dddddf} } }| d}|ddkrntjj| dd|dg} tjj| dd|dg} tjj|dd|dg}t| | |||||d|d||| o|dkd \}}}}| r| | | ||||||_ ||_ ||_ ||_ ||_||_||_|d d|f}| s|S|||fS) Nr rrrPrr8r9r:r;r<r=.) requires_gradrRdetachsizernn functionalpad_wrapped_flash_attn_forwardsave_for_backwardr6r7r8 window_sizer;r<rn)ctxqkvr6r7r8rr;r<rnr=is_grad_enabledis_gradr3r4r5 head_size_og out_paddedrGrHrIrFrrrforwardsB R    zFlashAttnQKVPackedFunc.forwardc Gs(|j\}}}}}}|jdddg|jddR} tj| |j|jd} |d} |} | ddkr@tjj |dd| dg} t | |||||| dddddf| dddddf| dddddf|j |j |j |jd|jd|j|j|j|d| d d|jd f} | dddddddddf S) NrxrQrrrrPrI.r ) saved_tensorsrRrrTrLrr|r}r~r_wrapped_flash_attn_backwardr6r7r8rr;r<rn) rrmargsr3r4r5rFrGrI qkv_shapedqkvr dout_paddedrrrbackwards:$  zFlashAttnQKVPackedFunc.backwardN__name__ __module__ __qualname__ staticmethodrrrrrrrvs  /rvc@ru)FlashAttnVarlenQKVPackedFuncc Csv| o|j} |dur|jdd}|dddf|dddf|dddf}}}|d}|ddkretjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t|||||||||||d|d|| | o}|dkdd\}}}}| r| |||||||||_ ||_ ||_ ||_ ||_||_| |_| |_|dd|f}| s|S|||fS) Nr rwrrrPrr8r9r:r;r<r=ra.)rzrRr{r|rr}r~r"_wrapped_flash_attn_varlen_forwardrr6 max_seqlenr7r8rr;r<rn)rr cu_seqlensrr6r7r8rr;r<rnr=rrr3r4r5rrrGrHrIrFrrrr sN @   z$FlashAttnVarlenQKVPackedFunc.forwardcGs(|j\}}}}}}} |jdddg|jddR} tj| |j|jd} |d} |} | ddkrAtjj |dd| dg} t | |||||| dddf| dddf| dddf|||j |j |j |j |j|jd|jd|j|j|j| d| d d|jd f} | dddddddddddf S) NrrxrQrPrrrr.r )rrRrrTrLrr|r}r~r#_wrapped_flash_attn_varlen_backwardrr6r7r8rr;r<rn)rrmrr3r4r5rFrGrrIrrrrrrrrCsB$  z%FlashAttnVarlenQKVPackedFunc.backwardNrrrrrr s  7rc@ru)FlashAttnKVPackedFuncc  Csj| o tdd||fD} |dur|jdd}|dddddf|dddddf} }|d}|ddkrhtjj|dd|dg}tjj| dd|dg} tjj|dd|dg}t|| |||||d|d||| o||dkd \}}}}| r| || ||||||_ ||_ ||_ ||_ ||_||_| |_|d d|f}| s|S|||fS) Ncs|]}|jVqdSr%rzrArrr x z0FlashAttnKVPackedFunc.forward..r rwrrrxrry.)anyrRr{r|rr}r~rrrr6r7r8rr;r<rn)rr3kvr6r7r8rr;r<rnr=rrr4r5rrrGrHrIrFrrrrisF 6    zFlashAttnKVPackedFunc.forwardcGs8|j\}}}}}}t|} |jdddg|jddR} tj| |j|jd} |d} |} | ddkrEtjj |dd| dg} t | |||||| | dddddf| dddddf|j |j |j|jd|jd|j|j|j|d| d d|jd f} | d d|jd f} | | dddddddddf S) NrrPrQrxrrrr.r )rrrSrRrTrLrr|r}r~rrr6r7r8rr;r<rn)rrmrr3r4r5rFrGrIrjkv_shapedkvrrrrrrs> $  zFlashAttnKVPackedFunc.backwardNrrrrrrh  2rc@ru)FlashAttnVarlenKVPackedFunccCsx|o tdd||fD}|dur|jdd}|dddf|dddf}}|d}|ddkrbtjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t|||||||||| | d| d| | |oz|dkdd \}}}}|r| ||||||||||_ ||_ ||_ ||_ | |_| |_| |_| |_| |_|d d|f}|s|S|||fS) Ncsrr%rrArrrrrz6FlashAttnVarlenKVPackedFunc.forward..r rwrrrPrr.)rrRr{r|rr}r~rrrr6r_r`r7r8rr;r<rn)rr3rr]r^r_r`r6r7r8rr;r<rnr=rrr4r5rrrGrHrIrFrrrrsX *   z#FlashAttnVarlenKVPackedFunc.forwardcGsD|j\}}}}}}} } t|} |jdddg|jddR} tj| |j|jd} |d}|}|ddkrGtjj |dd|dg}t ||||||| | dddf| dddf|| |j |j |j|j|j|jd|jd|j|j|j| d| dd|jd f} | dd|jd f} | | dddddddddddddfS) NrrPrQrrrr.r )rrrSrRrTrLrr|r}r~rrr_r`r6r7r8rr;r<rn)rrmrr3r4r5rFrGr]r^rIrjrrrrrrrrsF $  "z$FlashAttnVarlenKVPackedFunc.backwardNrrrrrrs  ?rc@ru) FlashAttnFuncc  Cs6| o tdd|||fD} |dur|jdd}|d}|ddkrNtjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t|||||||d|d|| | ob|dkd \}}}}| r|||||||||_ ||_ ||_ ||_ ||_ | |_| |_|d d|f}| s|S|||fS) Ncsrr%rrArrrr8rz(FlashAttnFunc.forward..r rwrxrrrry.)rrRr|rr}r~rrrr6r7r8rr;r<rn)rr3r4r5r6r7r8rr;r<rnr=rrrrrGrHrIrFrrrr(sD     zFlashAttnFunc.forwardcGs |j\}}}}}}t|t|t|} } } |d} |} | ddkr5tjj|dd| dg} t| |||||| | | |j|j |j |j d|j d|j |j |j|d| dd|jdf} | dd|jdf} | dd|jdf} | | | dddddddddf S)Nrxrrrr.r )rrrSr|r}r~rrr6r7r8rr;r<rnrR)rrmrr3r4r5rFrGrIrjrkrlrrrrrr[s<"  zFlashAttnFunc.backwardNrrrrrr'rrc@ru)FlashAttnVarlenFunccCsP|o tdd|||fD}| dur|jdd} |d}|ddkrNtjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t||||||||| | | d| d| | |of|dk|d \}}}}|r|||||||||||_ ||_ ||_ | |_ | |_ | |_| |_| |_||_|d d|f}|s|S|||fS) Ncsrr%rrArrrrrz.FlashAttnVarlenFunc.forward..r rwrPrrrr.)rrRr|rr}r~rrrr6r_r`r7r8rr;r<rn)rr3r4r5r]r^r_r`r6r7r8rr;r<rnr=rarrrrrGrHrIrFrrrr~sV    zFlashAttnVarlenFunc.forwardcGs&|j\}}}}}}} } t|t|t|} } } |d}|}|ddkr7tjj|dd|dg}t||||||| | | || |j|j |j |j |j |j d|j d|j|j|j| d| dd|jdf} | dd|jdf} | dd|jdf} | | | ddddddddddddddfS)NrPrrrr.r )rrrSr|r}r~rrr_r`r6r7r8rr;r<rnrR)rrmrr3r4r5rFrGr]r^rIrjrkrlrrrrrrsD"  &zFlashAttnVarlenFunc.backwardNrrrrrr}s  Arr r c Cs t|||||||||t S)adropout_p should be set to 0.0 during evaluation If Q, K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of Q, K, V. For multi-query and grouped-query attention (MQA/GQA), please see flash_attn_kvpacked_func and flash_attn_func. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i - window_size[0], i + window_size[1]] inclusive. Arguments: qkv: (batch_size, seqlen, 3, nheads, headdim) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rvapplyrr) rr6r7r8rr;r<rnreturn_attn_probsrrrflash_attn_qkvpacked_funcs-rc Cs"t|||||||||| t S)a dropout_p should be set to 0.0 during evaluation If K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of K, V. Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (batch_size, seqlen, nheads, headdim) kv: (batch_size, seqlen, 2, nheads_k, headdim) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr) r3rr6r7r8rr;r<rnrrrrflash_attn_kvpacked_func!s?rc C$t|||||||||| | t S)a dropout_p should be set to 0.0 during evaluation Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (batch_size, seqlen, nheads, headdim) k: (batch_size, seqlen, nheads_k, headdim) v: (batch_size, seqlen, nheads_k, headdim) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr) r3r4r5r6r7r8rr;r<rnrrrrflash_attn_funcos=rc Cr)a dropout_p should be set to 0.0 during evaluation If Q, K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_varlen_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of Q, K, V. For multi-query and grouped-query attention (MQA/GQA), please see flash_attn_varlen_kvpacked_func and flash_attn_varlen_func. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i - window_size[0], i + window_size[1]] inclusive. Arguments: qkv: (total, 3, nheads, headdim), where total = total number of tokens in the batch. cu_seqlens: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into qkv. max_seqlen: int. Maximum sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr) rrrr6r7r8rr;r<rnrrrr flash_attn_varlen_qkvpacked_funcs2rcCs*t|||||||||| | | | | tS)a dropout_p should be set to 0.0 during evaluation If K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of K, V. Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. kv: (total_k, 2, nheads_k, headdim), where total_k = total number of key tokens in the batch. cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr)r3rr]r^r_r`r6r7r8rr;r<rnrrrrflash_attn_varlen_kvpacked_funcs"IrcCs.t|||||||||| | | | | ||tS)aq dropout_p should be set to 0.0 during evaluation Supports multi-query and grouped-query attention (MQA/GQA) by passing in K, V with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr)r3r4r5r]r^r_r`r6r7r8rr;r<rnrrarrrflash_attn_varlen_funcZs&IrT cache_seqlenscache_batch_idx cache_leftpadcCs|ddks Jd|ddksJddd|||fD\}}}| dur.|jdd} |durJt|trJtj|jd f|tj|jd }t|}t|}t| } t |||||||||| | |d| | | d | d|||\}}|rv||fS|S) a< If k and v are not None, k_cache and v_cache will be updated *inplace* with the new values from k and v. This is useful for incremental decoding: you can pass in the cached keys/values from the previous step, and update them with the new keys/values from the current step, and do attention with the updated cache, all in 1 kernel. If you pass in k / v, you must make sure that the cache is large enough to hold the new values. For example, the KV cache could be pre-allocated with the max sequence length, and you can use cache_seqlens to keep track of the current sequence lengths of each sequence in the batch. Also apply rotary embedding if rotary_cos and rotary_sin are passed in. The key @k will be rotated by rotary_cos and rotary_sin at indices cache_seqlens, cache_seqlens + 1, etc. If causal or local (i.e., window_size != (-1, -1)), the query @q will be rotated by rotary_cos and rotary_sin at indices cache_seqlens, cache_seqlens + 1, etc. If not causal and not local, the query @q will be rotated by rotary_cos and rotary_sin at indices cache_seqlens only (i.e. we consider all tokens in @q to be at position cache_seqlens). See tests/test_flash_attn.py::test_flash_attn_kvcache for examples of how to use this function. Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Note: Does not support backward pass. Arguments: q: (batch_size, seqlen, nheads, headdim) k_cache: (batch_size_cache, seqlen_cache, nheads_k, headdim) if there's no block_table, or (num_blocks, page_block_size, nheads_k, headdim) if there's a block_table (i.e. paged KV cache) page_block_size must be a multiple of 256. v_cache: (batch_size_cache, seqlen_cache, nheads_k, headdim) if there's no block_table, or (num_blocks, page_block_size, nheads_k, headdim) if there's a block_table (i.e. paged KV cache) k [optional]: (batch_size, seqlen_new, nheads_k, headdim). If not None, we concatenate k with k_cache, starting at the indices specified by cache_seqlens. v [optional]: (batch_size, seqlen_new, nheads_k, headdim). Similar to k. rotary_cos [optional]: (seqlen_ro, rotary_dim / 2). If not None, we apply rotary embedding to k and q. Only applicable if k and v are passed in. rotary_dim must be divisible by 16. rotary_sin [optional]: (seqlen_ro, rotary_dim / 2). Similar to rotary_cos. cache_seqlens: int, or (batch_size,), dtype torch.int32. The sequence lengths of the KV cache. cache_batch_idx: (batch_size,), dtype torch.int32. The indices used to index into the KV cache. If None, we assume that the batch indices are [0, 1, 2, ..., batch_size - 1]. If the indices are not distinct, and k and v are provided, the values updated in the cache might come from any of the duplicate indices. cache_leftpad: (batch_size,), dtype torch.int32. The index that the KV cache starts. If None, assume 0. block_table [optional]: (batch_size, max_num_blocks_per_seq), dtype torch.int32. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. rotary_interleaved: bool. Only applicable if rotary_cos and rotary_sin are passed in. If True, rotary embedding will combine dimensions 0 & 1, 2 & 3, etc. If False, rotary embedding will combine dimensions 0 & rotary_dim / 2, 1 & rotary_dim / 2 + 1 (i.e. GPT-NeoX style). alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. num_splits: int. If > 1, split the key/value into this many chunks along the sequence. If num_splits == 1, we don't split the key/value. If num_splits == 0, we use a heuristic to automatically determine the number of splits. Don't change this unless you know what you are doing. return_softmax_lse: bool. Whether to return the logsumexp of the attention scores. Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). r rz+k_cache must have contiguous last dimensionz+v_cache must have contiguous last dimensioncSr?rr@rArrrrC&rDz+flash_attn_with_kvcache..NrwrrQ) r rR isinstanceintrfullZint32rrrEZ fwd_kvcache)r3Zk_cacheZv_cacher4r5Z rotary_cosZ rotary_sinrrrrar7r8rr;Zrotary_interleavedr<Z num_splitsZreturn_softmax_lserFrGrrrflash_attn_with_kvcachesFlrr%) r r r\NFNNNF)NF)r\NFrr\NFF) r\NFrr\NFFN)NNNNNNNNNFrr\TNrF);typingrrrrrZtorch.nnr}osgetenvZUSE_TRITON_ROCMZflash_attn_triton_amdr rEZflash_attn_2_cudarrr! __version__ZlibraryZ custom_opZ_torch_custom_op_wrapperZ register_fakeZ_torch_register_fake_wrapperr-r2ZTensorfloatboolrrJr[opsZ flash_attnrrerirrprrrrsrtrZautogradFunctionrvrrrrrrrrrrrrrrrrs                      2       %        3      !        ?      ( R^WhVk > R Q I d b