o )i@sddlmZddlZdejdedejfddZdejded ed edejf d d Z dd eejdeejded ed edeedeejfddZddejfddZej fdejdeej dejfddZ dS))OptionalNkey_padding_mask batched_dimreturncCs0|jdksJ|\}}||}t||||S)N)ndimsize_reshape_key_padding_mask)rr batch_sizesrc_len num_headsr o/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/xformers/components/attention/utils.pyreshape_key_padding_mask s rr r r cCs@|j||fks J||dd|d|dd||d|}|S)N)shapeviewexpandZreshape)rr r r r r rr s  r att_masktgt_lencCs~|dur|}|dur=|j||fksJt||||}|dur'|d|d}|S|jtjkr4||}|S||td}|S)Nr-inf) rr rdtypetorchbool logical_andZ masked_fillfloat)rrr r r rr r rmaybe_merge_masks%s rF softmax_matcCstj|d|j|jd}|}|r$dttj|dd|dd}ndtjtj|ddddjddddf|dd}t |D]$}t ||}t d|d|t |d |t |d ||}qD|S) z Computing the Moore-Penrose inverse. Use an iterative method from (Razavi et al. 2014) to approximate the Moore-Penrose inverse via efficient matrix-matrix multiplications. r)devicerr)dimNg? ) reyerr rmaxsumZ transposevaluesrangematmul)rZn_iterZpinverse_original_initikv_kvr r riterative_pinvAs&&&   *r1maskrcCs4|jtjks Jdtj||d}td||<|S)NzEThis util is meant to convert in between bool masks and additive ones)rr)rrrZ zeros_liker)r2rZmask_r r rbool_mask_to_additivecs r3)N)rF) typingrrZTensorintrr rr1Zfloat32rr3r r r rsZ     #