o )iZ@sdZddlZddlmZddlmZddlmZmZddl Z ddl m Z m Z ddl mZddlmZmZmZdd lmZdd lmZdd lmZd d lmZd dlmZeeZeGdddZdee efde e!e!fde!fddZ"dS)a Expert parallelism load balancer (EPLB) metrics and states. # Glossary - **Logical Expert**: An expert that is part of the model's logical structure. It holds a set of weights and is replicated across multiple physical experts. - **Redundant Expert**: To achieve load balancing, for some popular logical experts, we create additional copies of the expert weights. During inference, each of these copies can be routed to by the same set of tokens. - **Physical Expert**: An expert that is instantiated on a specific device. It is a replica of a logical expert and can be rearranged across devices. I.e., one logical expert may have multiple sets of weights initialized on different devices, and each of these sets is a physical expert. - **Local Physical Expert**: A physical expert that is instantiated on the current device. For example: DeepSeek-R1 has 256 logical experts, so each MoE layer has 256 sets of linear layer weights in the model parameters. If we add 32 redundant experts, DeepSeek-R1 will have 256 + 32 = 288 physical experts in total. And when deploying, we'll have 288 sets of linear layer weights for each MoE layer. If we have 32 EP ranks, then each GPU will hold 288 / 32 = 9 local physical experts. N)Sequence) dataclass)OptionalUnion) ProcessGroup all_reduce)ParallelConfig) get_ep_groupget_node_countin_the_same_node_as)StatelessProcessGroup) init_logger)MixtureOfExperts)rebalance_experts) rearrange_expert_weights_inplacec@sreZdZUdZejed< ejed< ejed< ejed< ejed< dZeed< dZ eed < dZ eed < dZ eed < e d ed ede efddZe   d&dedejdedeejdeejdeeeefddfddZ   d'dededededdf ddZ   d(deded!edeejdeeeefddf d"d#Ze deejejffd$d%ZdS)) EplbStatez EPLB metrics.physical_to_logical_maplogical_to_physical_maplogical_replica_countexpert_load_passexpert_load_windowrexpert_load_window_stepexpert_load_window_sizeexpert_rearrangement_step"expert_rearrangement_step_intervalnum_routed_expertsnum_redundant_expertsreturncs*tt}|fddt|D7}|S)aj Build an initial expert arrangement using the following structure: [original routed experts, redundant experts] Returns: physical_to_logical_map (Sequence[int]): A list of integers, where each integer is the index of the logical expert that the corresponding physical expert maps to. csg|]}|qSr).0irrl/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/distributed/eplb/eplb_state.py szJEplbState.build_initial_global_physical_to_logical_map..)listrange)rrZglobal_physical_to_logical_maprr"r#,build_initial_global_physical_to_logical_maps  z6EplbState.build_initial_global_physical_to_logical_mapNmodeldeviceparallel_configglobal_expert_loadold_global_expert_indices rank_mappingc Cs~||j|j}tj||d}d} |j| ks!Jd|jd| | d} tj|j| fd|d} tj|jf|tjd} t |j D]} || }| | || |f<| |d7<q@| d |j d}| d |j dd} | d |j d} tj|j |j ftj|d }|j}tj||j |j ftj|d }|j}td||d }|d urtj}|j|j |jfksJ|jtjksJ|j }|j}t}|}||dkrd}td |d |t|||||\}}}|jd}|| jdksJtjjj |d| jd|fdd}|!|}| "|| "||#|| | |d ur3t$|||j%|d|d}||| | |||||dS)z/ Build the initial EPLB state. )r)iznum_redundant_experts z must be less than or equal to r)r)dtyperr/r)NTnum_gpus % num_nodes != 0, not using hierarchical rearrangement algorithm. num_gpus= , num_nodes=valueF)rrr)&r'rrtorchtensorfullnum_logical_expertszeroslongr&num_physical_experts unsqueezeexpandnum_moe_layers contiguousint32Zeplb_window_sizeeplb_step_intervalmaxr device_groupshaper/int64num_expert_groupsr sizelogger warning_oncernn functionalpadtocopy_Zset_eplb_staterexpert_weights)clsr(r)r*r+r,r-Zphysical_to_logical_map_listrZMAX_EXPERT_REDUNDANCYZmax_slots_per_logical_expertrrr!Z logical_idxrrrrBrep_group num_replicas num_groups num_nodesnum_gpusnew_physical_to_logical_mapnew_logical_to_physical_mapnew_logical_replica_countmax_physical_slotsrrr#builds                 zEplbState.buildFis_dummy is_profile log_statscCsP|r |j|dddS|r|j|ro|j}tj}t||d||jd| dj dd }|j ddj dd}|j ddjj dd} t|| g} | \} } | dkr_| | nd} |dkrotd | | | |s|j|j|j<|jd 7_|j|jkrd|_|j|jd 7_|j|jkrd|_||dSdS) aD Step the EPLB state. Args: model (MixtureOfExperts): The MoE model. is_dummy (bool): If `True`, this is a dummy step and the load metrics recorded in this forward pass will not count. Defaults to `False`. is_profile (bool): If `True`, perform a dummy rearrangement with maximum communication cost. This is used in `profile_run` to reserve enough memory for the communication buffer. log_stats (bool): If `True`, log the expert load metrics. # Stats The metrics are all summed up across layers. - `avg_tokens`: The average load across ranks. - `max_tokens`: The maximum load across ranks. - `balancedness`: The ratio of average load to maximum load. T)r]Ngrouprr.dimgzsz&EplbState.rearrange..r2r3r4z$Rearranged experts%sin %.2f seconds.z (profile)  )1r rDrkr6cudaZ synchronizetime perf_counterrIrlr:rr?r9rr/r)Z scatter_add_rr=Z expand_asr;r7rErA distributed broadcast cpu_grouprerr<rGlenrH_node_count_with_rank_mappingrhr rUrJrrrPrNrOrrKrLrMr)rmr(r]ror+r-rRZep_rankZ time_startZ is_main_rankZlogical_expert_load_windowmetadataZglobal_expert_load_windowr,rSrTr|rUrVrWrXrYrZZtime_endrrr#rcs                   zEplbState.rearrangecCst}tjdtjdd}tjj||jdd|\}}}tj||ftj |j d}t ||j dtj||ftj |j d}tjj||j dd||fS)zQ Receive the expert load and old placement from the master rank. rsr0rrtr_) r r6emptyrArzr{r|rjr:rFr)rrD)rRrr?r9Znum_old_physical_expertsr+r,rrr# recv_state(s0zEplbState.recv_state)NNN)FFF)FTNN)__name__ __module__ __qualname____doc__r6ZTensor__annotations__rintrrr staticmethodrr' classmethodrr)rrdictr[boolrnrctuplerrrrr#r2s             U  rpgr-rc Cst|tr tjj|d}n|j}|dkrdSdg|}d}t|D]8}||dkr*q!||vs0J||dkr7q!|d7}|||<t||}t|D]\}}|rX||dkrX|||<qHq!|S)Nr_rrr.) isinstancerr6rzZget_world_size world_sizer&r enumerate) rr-rZnode_assignmentZ next_node_idZ current_rankZsame_node_flagsZ other_rankZ is_same_noderrr#r~Fs,       r~)#rrxcollections.abcr dataclassesrtypingrrr6Ztorch.distributedrrZ vllm.configrZvllm.distributed.parallel_stater r r Zvllm.distributed.utilsr Z vllm.loggerr Z%vllm.model_executor.models.interfacesrZrebalance_algorZrebalance_executerrrIrrrr~rrrr#s8