o |iK @sddlZddlmZddlmmZddlmZmZm Z ddl m Z ddl m Z ddZGdddejZGd d d ejZGd d d ejZGd ddejZGdddejZdejdedeejeeefffddZdejdedeeefdeeefdejf ddZdededejdejfddZd ejd!ejd"ejdeeefdeeefdejf d#d$ZGd%d&d&ejZd+d'd(Z d+d)d*ZdS),N)OptionalTupleType)partial)flash_attn_qkvpacked_funccCsj|j}|d}||kr3|dddd}|tj}tj|||fdddd|}|dddd}|S|S) NrbicubicTF)sizemode antialias align_corners)dtyper permutetotorchfloat32F interpolate)Zabs_posZtgt_sizersrc_sizeZ old_pos_embedZ new_pos_embedrY/home/app/DeepSeek-OCR/DeepSeek-OCR-master/DeepSeek-OCR-vllm/deepencoder/sam_vary_sdpa.py get_abs_poss"  rc sPeZdZejfdededeejddffdd Zde j de j fd d Z Z S) MLPBlock embedding_dimmlp_dimactreturnNcs2tt|||_t|||_||_dSN)super__init__nnLinearlin1lin2r)selfrrr __class__rrr!,s  zMLPBlock.__init__xcCs||||Sr)r%rr$r&r)rrrforward7szMLPBlock.forward) __name__ __module__ __qualname__r"GELUintrModuler!rTensorr+ __classcell__rrr'rr+s rcsBeZdZd dededdffdd Zdejdejfd d ZZ S) LayerNorm2dư> num_channelsepsrNcs8ttt||_tt||_||_ dSr) r r!r" Parameterronesweightzerosbiasr7)r&r6r7r'rrr!>s  zLayerNorm2d.__init__r)cCsn|jddd}||djddd}||t||j}|jddddf||jddddf}|S)NrT)keepdimr )meanpowrsqrtr7r:r<)r&r)usrrrr+Ds ,zLayerNorm2d.forward)r5) r,r-r.r0floatr!rr2r+r3rrr'rr4=sr4c#seZdZdddddddddejejdd dd d fd ed edededededededede ej de ej dedededede edfddf"fdd Z d!e jde jfd"d#ZZS)$ImageEncoderViTr @TFrrimg_size patch_sizein_chans embed_dimdepth num_heads mlp_ratio out_chansqkv_bias norm_layer act_layer use_abs_pos use_rel_posrel_pos_zero_init window_sizeglobal_attn_indexes.rNcst||_t||f||f||d|_d|_| r*tt d||||||_t |_ t |D]"}t |||| | | | |||vrD|nd||||fd }|j |q3ttj||dddt|tj||dddd t||_tjd d dd ddd |_tjd ddd ddd |_dS)a Args: img_size (int): Input image size. patch_size (int): Patch size. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. depth (int): Depth of ViT. num_heads (int): Number of attention heads in each ViT block. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool): If True, add a learnable bias to query, key, value. norm_layer (nn.Module): Normalization layer. act_layer (nn.Module): Activation layer. use_abs_pos (bool): If True, use absolute positional embeddings. use_rel_pos (bool): If True, add relative positional embeddings to the attention map. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. window_size (int): Window size for window attention blocks. global_attn_indexes (list): Indexes for blocks using global attention. ) kernel_sizestriderMrNNrr) dimrPrQrSrTrUrWrXrY input_sizeF)r[r<r)r[paddingr<rJir )r[r\r_r<rE)r r!rK PatchEmbed patch_embed pos_embedr"r8rr; ModuleListblocksrangeBlockappend SequentialConv2dr4necknet_2net_3)r&rKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZiblockr'rrr!Ns^ %   zImageEncoderViT.__init__r)cCsl||}|jdur|t|j|d}|jD]}||}q||dddd}||}||}|S)Nrrrr ) rarbrr rdrjrrkrl)r&r)blkZ neck_outputZ conv2_outputZ conv3_outputrrrr+s      zImageEncoderViT.forward)r,r-r.r" LayerNormr/r0rCboolrr1rr!rr2r+r3rrr'rrDMsj      [rDcseZdZdZddejejddddfdeded ed e d e ej d e ej d e de dede e eefddffdd ZdejdejfddZZS)rfzSTransformer blocks with support of window attention and residual propagation blocksrITFrNr]rPrQrSrTrUrWrXrYr^rc sft|||_t|||||| dkr| n| | fd|_|||_t|t|||d|_| |_ dS)ai Args: dim (int): Number of input channels. num_heads (int): Number of attention heads in each ViT block. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool): If True, add a learnable bias to query, key, value. norm_layer (nn.Module): Normalization layer. act_layer (nn.Module): Activation layer. use_rel_pos (bool): If True, add relative positional embeddings to the attention map. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. window_size (int): Window size for window attention blocks. If it equals 0, then use global attention. input_size (tuple(int, int) or None): Input resolution for calculating the relative positional parameter size. r)rPrSrWrXr^)rrrN) r r!norm1 Attentionattnnorm2rr0mlprY) r&r]rPrQrSrTrUrWrXrYr^r'rrr!s    zBlock.__init__r)cCs|}||}|jdkr|jd|jd}}t||j\}}||}|jdkr3t||j|||f}||}||||}|S)Nrrr )rrrYshapewindow_partitionrtwindow_unpartitionrvru)r&r)shortcutHWpad_hwrrrr+s    z Block.forward)r,r-r.__doc__r"rpr/r0rCrqrr1rrr!rr2r+r3rrr'rrfsD    ,rfcsleZdZdZ     ddededed ed ed eeeefd dffd d Zde j d e j fddZ Z S)rsz=Multi-head Attention block with relative position embeddings.TFNr]rPrSrWrXr^rcst||_||}|d|_tj||d|d|_t|||_||_|jrS|dus1Jdt t d|dd||_ t t d|dd||_ dSdS) a Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. qkv_bias (bool): If True, add a learnable bias to query, key, value. rel_pos (bool): If True, add relative positional embeddings to the attention map. rel_pos_zero_init (bool): If True, zero initialize relative positional parameters. input_size (tuple(int, int) or None): Input resolution for calculating the relative positional parameter size. gr)r<NzBInput size must be provided if using relative positional encoding.r rr)r r!rPscaler"r#qkvprojrWr8rr; rel_pos_h rel_pos_w)r&r]rPrSrWrXr^head_dimr'rrr!s    $zAttention.__init__r)c Cs|j\}}}}|||||d|jdddddd}|d||j||dd\}}} d\} } |jrGt||j|j ||f||f\} } | ||j||d}| ||j||d}| ||j||d} |jr| ||j| d| d| d} | ||j| d| d| d} | |  ||j| d| d| d} t j jj||| | d}n t j j||| }| ||j||dddddd|||d}||}|S) Nrr rr)NN) attn_mask)rwrreshaperPrunbindrWadd_decomposed_rel_posrrviewr rr" functionalscaled_dot_product_attentionr) r&r)Br{r|_rqkvrel_hrel_w attn_biasrrrr+#s$,& &&,. zAttention.forward)rTFTN) r,r-r.r~r0rqrrr!rr2r+r3rrr'rrss.$rsr)rYrc Cs|j\}}}}||||}||||}|dks|dkr+t|ddd|d|f}||||}} |||||| |||}|ddddddd|||} | || ffS)aU Partition into non-overlapping windows with padding if needed. Args: x (tensor): input tokens with [B, H, W, C]. window_size (int): window size. Returns: windows: windows after partition with [B * num_windows, window_size, window_size, C]. (Hp, Wp): padded height and width before partition rrrr rr)rwrpadrr contiguous) r)rYrr{r|Cpad_hpad_wHpWpwindowsrrrrxFs $ rxrr}hwc Cs|\}}|\}}|jd||||}||||||||d} | dddddd|||d} ||ks=||krO| ddd|d|ddf} | S) a Window unpartition into original sequences and removing padding. Args: windows (tensor): input tokens with [B * num_windows, window_size, window_size, C]. window_size (int): window size. pad_hw (Tuple): padded height and width (Hp, Wp). hw (Tuple): original height and width (H, W) before padding. Returns: x: unpartitioned sequences with [B, H, W, C]. rrrrr rrN)rwrrr) rrYr}rrrr{r|rr)rrrry^s$$ryq_sizek_sizerel_posc Cstdt||d}|jd|kr>|j}|tj}tj| d|jdd ddd|dd|}| d| dd}n|}tj ||j ddddft||d }tj ||j ddddft||d }|||dt||d }|| S) a\ Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (int): size of query q. k_size (int): size of key k. rel_pos (Tensor): relative position embeddings (L, C). Returns: Extracted positional embeddings according to relative positions. r rrrlinear)r r )deviceNg?)r0maxrwrrrrrrrrarangerlong) rrrZ max_rel_distrZrel_pos_resizedZq_coordsZk_coordsrelative_coordsrrr get_rel_posws"  ** rrrrcCs|\}}|\}}t|||} t|||} |j\} } } || ||| }td|| }td|| }|d}|d}|| |||d}|| ||d|}||fS)a Calculate decomposed Relative Positional Embeddings from :paper:`mvitv2`. https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py # noqa B950 Args: q (Tensor): query q in the attention layer with shape (B, q_h * q_w, C). rel_pos_h (Tensor): relative position embeddings (Lh, C) for height axis. rel_pos_w (Tensor): relative position embeddings (Lw, C) for width axis. q_size (Tuple): spatial sequence size of query q with (q_h, q_w). k_size (Tuple): spatial sequence size of key k with (k_h, k_w). Returns: attn (Tensor): attention map with added relative positional embeddings. zbhwc,hkc->bhwkzbhwc,wkc->bhwkrr)rrwrreinsum unsqueeze)rrrrrq_hq_wk_hk_wRhRwrrr]r_qrrrrrrs     rc steZdZdZ     ddeeefdeeefdeeefd ed ed d f fd d Zdejd ejfddZ Z S)r`z# Image to Patch Embedding. rFrFrrrrGr[r\r_rMrNrNcs$ttj|||||d|_dS)aP Args: kernel_size (Tuple): kernel size of the projection layer. stride (Tuple): stride of the projection layer. padding (Tuple): padding size of the projection layer. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. )r[r\r_N)r r!r"rir)r&r[r\r_rMrNr'rrr!s   zPatchEmbed.__init__r)cCs||}|dddd}|S)Nrr rr)rrr*rrrr+s zPatchEmbed.forward)rrrrrG) r,r-r.r~rr0r!rr2r+r3rrr'rr`s*   r`cCstdddgd|dS)NrGrH)r rr )encoder_embed_dim encoder_depthencoder_num_headsencoder_global_attn_indexes checkpoint) _build_sam)rrrrbuild_sam_vit_bsrc Cszd}d}d}||}t|||dttjjdd||dd|d|d } |dur;t|} | jd d | Ddd t|| S) NrJrErFrr5)r7T) rOrNrKrQrTrPrLrSrWrZrYrRcSs&i|]\}}d|vr|dd|qS)Zvision_tower_highNr).0rrrrr s&z_build_sam..)strict) rDrrr"rploadload_state_dictitemsprint) rrrrrZprompt_embed_dim image_sizeZvit_patch_sizeZimage_embedding_sizeZ image_encoder state_dictrrrrs. rr)rtorch.nnr"torch.nn.functionalrrtypingrrr functoolsr flash_attnrrr1rr4rDrfrsr2r0rxryrrr`rrrrrrsT   mB*J   #   % "