o )i:@sdZddlmZmZmZddlmZddlmZddl Z ddl m Z m Z m Z mZmZdeded ed efd d Zd eded ed efddZd eded efddZdededeedeed eeeeeff ddZded edeedeedee jdee jde d dfddZ  d)de jde jdeee jde d ed!eeeefd dfd"d#Zde jd!eeefd$ed e jfd%d&Zde jd!eeefd e jfd'd(Zd#gZdS)*zh The actual execution of the rearrangement. This involves the exchange of expert weights between GPUs. )IterableMutableSequenceSequence)partial)OptionalN)P2POp ProcessGroup all_gatherbatch_isend_irecvget_global_rank local_idx local_cntep_rankreturncCs |||S)z@ Convert a local expert index to a global expert index. )r r rrrs/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/distributed/eplb/rebalance_execute.pyidx_local_to_global r global_idxcCs |||S)z@ Convert a global expert index to a local expert index. r)rr rrrridx_global_to_localrrcCs||S)z8 Convert a global expert index to a rank index. r)rr rrrglobal_idx_to_rank(sridxnum_local_experts old_indices new_indicesc stt|d}g}g}t|D]\}}||kr'||} |r"|d| kr'|| qt|D]\}}||krE||} |r@|d| krE|| q,t|fdd|D} || fS)a Get the ranks of the experts that need to be exchanged. Args: idx: The index of the expert. num_local_experts: The number of local experts. old_indices: The old indices of the experts. new_indices: The new indices of the experts. Returns: A tuple of two lists: - The ranks of the experts that need to be sent. - The ranks of the experts that need to be received. )r csg|]}|vr|qSrr).0rankZranks_to_send_setrr \sz,get_ep_ranks_with_expert..)rr enumerateappendset) rrrrZ global2rank ranks_to_send ranks_to_recvierZranks_to_recv_actualrrrget_ep_ranks_with_expert2s.   r'expert_weightsexpert_weights_bufferep_groupcsVtt||dfddt|D}|dd}t|D]Bt|D]7|r3q(dks?dkr@q(kr_d|<t||D] \} } | | qQq(qg} i} t|D]} | dkrvqi| | vr{qi| | <qit| D]U\} t| |\}}t|t|}| |}||}||}|||}t||}||}|t|kr| |||D]t || fdd|D7} qqi}t|D]|rq} | dkrq| |vrq|| <qt|D]D\} t| |\}}t|t|}| |}t||}||kr0|||n|||t || fdd|D7} q| rZt | }|D]}| qRt|D]J|rgq^|rt||D]\} } | | qqq^} | dkrq^|| t||D]\} } | | qq^dS) z< Perform expert weights rearrangement of one layer. )r rcs$g|]}||kqSrr)rr%) local2globalrrrrrvsz!shuffle_layer..NrTcs g|] }ttjj|qSr)rtorch distributedZisendrweight) dst_globalsrcrrrcs g|] }ttjj|qSr)rr,r-Zirecvr.)dst src_globalrrrr2)rrrangezipZcopy_sorteditemsr'lenindexr!r r wait)rrrrr(r)r*Z is_unchangedZis_received_locallyr/bufferZp2p_opsZexperts_send_locZexpertr#r$Znum_dst_per_senderZ sender_posZ recv_beginZrecv_endZ recv_ranksZremainder_startZ recver_posZexperts_recv_locreqsreqr)r3r0r+rrr1r4r shuffle_layercs                               r?Fold_global_expert_indicesnew_global_expert_indices is_profile rank_mappingc sR|durt||krt||}nt|||}|jd|jdks&J|j\}}t||ks3Jtt|djd}|j||fksGJ|} |} || |ksWJdd|dD} |rt|d| D]\} fddt | D} t j t | | |dqidSt |D]}t jt|| ||||||| |qdS)a Rearranges the expert weights in place according to the new expert indices. The value of the indices arguments are logical indices of the experts, while keys are physical. Args: old_global_expert_indices: Shape (num_moe_layers, num_physical_experts). new_global_expert_indices: Shape (num_moe_layers, num_physical_experts). expert_weights: A sequence of shape (num_moe_layers)(weight_count) of tensors of shape (num_local_physical_experts, hidden_size_i). For example, a linear layer may have up and down projection, so weight_count = 2. Each weight's hidden size can be different. ep_group: The device process group for expert parallelism. is_profile (bool): If `True`, do not perform any actual weight copy. This is used during profile run, where we only perform dummy communications to reserve enough memory for the buffers. rank_mapping: A dictionary mapping old rank to new rank. NrcSsg|]}t|qSr)r,Z empty_like)rwrrrr1sz4rearrange_expert_weights_inplace..csg|]}qSrr)r_r<rrr8s)group)r9size)_map_new_expert_indices_with_rank_mapping)_map_old_expert_indices_with_rank_mappingshapenextiterrr6r5r,r-Zbarrierr cudaZ synchronizer?tolist)r@rAr(r*rBrCZnum_moe_layersZnum_physical_expertsnum_local_physical_expertsrZep_sizer)r/Zdummy_recv_bufferlayerrrGr rearrange_expert_weights_inplacesd       rS new_ep_sizecCs|j\}}|s Jdt|}||}||}tj||fd|j|jd}t|D]9} || } | dura| dkra| |kra| |} | d|} | |} | d|}|dd| | f|dd| |f<q(|S)a Map the old global expert indices to the new global expert indices. Args: old_global_expert_indices: Shape (num_layers, old_ep_size * num_local_physical_experts). rank_mapping: Mapping from old rank to new rank. new_ep_size: New expert parallelism size. Returns: Mapped expert indices with shape (num_layers, new_ep_size * num_local_physical_experts). Rank mapping is requiredrZ fill_valuedtypedeviceNrrD)rLr9r,fullrWrXr5get)r@rCrT num_layersold_num_physical_experts old_ep_sizerQnew_num_physical_expertsmapped_expert_indicesold_ranknew_rank old_start_idx old_end_idx new_start_idx new_end_idxrrrrKRs,      rKcCs|j\}}|s Jdt|}tdd|D}||}||}tj||fd|j|jd}t|D]4} || } | dkrg| |krg| |} | d|} | |} | d|}|dd| |f|dd| | f<q3|S)NrUcss|]}|dkVqdS)rNr)rrarrr sz<_map_new_expert_indices_with_rank_mapping..rrVrrD) rLr9sumvaluesr,rYrWrXr5)rArCr[r^r]rTrQr\r_r`rarbrcrdrerrrrJs.     rJ)FN)__doc__collections.abcrrr functoolsrtypingrr,Ztorch.distributedrrr r r intrrrtupler'ZTensorr?booldictrSrKrJ__all__rrrrs         1   ^  4  "