o )i"@sdZddlZdejdedeejejffddZdejdedeejejejffd d Zdejd ed ed edef ddZdejded ed ededeejejejff ddZdgZ dS)aA Expert parallelism load balancer (EPLB) for vLLM. This module implements the core rearrangement algorithm. The rearrangement algorithm is adapted from [DeepSeek EPLB](https://github.com/deepseek-ai/eplb). Please find at [#12](https://github.com/deepseek-ai/EPLB/issues/12) an example on how the EPLB algorithm works. Nweight num_packsreturnc sL|j\}}||dks J||dkr2tj|dtj|jd|j}tj|tjd}||fS|j dddj }tj |dtjdd }tj |dd }t |D]N}dg|}dg|||D]=} tfd d t |D|jd } | ks~J| ||| f<| ||| f<|| ||| f7<| d7<qcqS||fS)a Pack n weighted objects to m packs, such that each bin contains exactly n/m objects and the weights of all packs are as balanced as possible. Parameters: weight: [X, n], the weight of each item num_packs: number of packs Returns: pack_index: [X, n], the pack index of each item rank_in_pack: [X, n], the rank of the item in the pack rdtypedevice)rT)Z descendingcpu) fill_valuerr )r c3s |] }|kr|VqdS)N).0iZgroups_per_packZ pack_itemsr p/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/distributed/eplb/rebalance_algo.py 6s z#balanced_packing..)key)shapetorcharangesizeint64r expandZ zeros_likefloatsortindicesr Z full_likerangemin __getitem__) rr num_layers num_groups pack_index rank_in_packrrZ pack_weightsgrouppackr rrbalanced_packingsH         r%num_phyc Cs|j\}}||}|dksJ|j}tj|tj|d|d}tj||tj|d}tj||tj|d}tj|tj|d} t||D])} ||j ddj } | |dd| f<|| | f|dd| f<|| | fd7<qA|||fS)a Replicate `num_log` experts to `num_phy` replicas, such that the maximum load of all replicas is minimized. Parameters: weight: [X, num_log] num_phy: total number of experts after replication Returns: phy2log: [X, num_phy], logical expert id of each physical expert rank: [X, num_phy], the replica rank logcnt: [X, num_log], number of replicas for each logical expert rrrr)dimN) rr rrrrepeatZzerosZonesrmaxr) rr&nZnum_logZ num_redundantr phy2logZranklogcntZarangenrZredundant_indicesr r rreplicate_expertsBs$    r-num_physical_expertsr num_nodesnum_gpuscCs|j\}}||dks J||}||dksJ||}||dks%J||dks-J||} dtjdtjfdd} |d||fd} t| |\} } | || |dtj|tj| j d d}| |}| d| d||}t |||\}}}|| d|}t|||\}}|| |}| |}| d|}| ||dtjd|||| j d  d dd  d}| d|}| d| |d}| |d d|}|||fS) aK Parameters: weight: [num_moe_layers, num_logical_experts] num_physical_experts: number of physical experts after replication num_groups: number of expert groups num_nodes: number of server nodes, where the intra-node network (e.g, NVLink) is faster num_gpus: number of GPUs, must be a multiple of `num_nodes` Returns: physical_to_logical_map: [num_moe_layers, num_physical_experts] logical_to_physical_map: [num_moe_layers, num_logical_experts, X] logical_count: [num_moe_layers, num_logical_experts] rpermrc Ss:t|}|d|tj|dtj|jd|j|S)Nrr) rZ empty_likescatter_rrrr rr)r1invr r rinverses z/rebalance_experts_hierarchical..inverserr)r r)rrTensorZ unflattensumr%Z unsqueezerrr flattengatherviewr-)rr.r r/r0rnum_logical_expertsZ group_sizeZgroups_per_nodeZphy_experts_per_gpur4Ztokens_per_groupZgroup_pack_indexZgroup_rank_in_packZlog2mlogZmlog2logZtokens_per_mlogZphy2mlogphyrankZmlogcntZtokens_per_phyr!r"Zphy2pphyZpphy2phyZ pphy2mlogZpphy2logZpphyrankr,r r rrebalance_experts_hierarchicalcsl         r= num_replicasc Cs|j\}}|}||dkrt|||||\}}} n t||dd|\}}} ||} | d} tj||| fdtj| jd} | |d d|| |tj |tj| jd |d|| | fS)aJ Entry point for expert-parallelism load balancer. Parameters: weight: [layers, num_logical_experts], the load statistics for all logical experts num_replicas: number of physical experts, must be a multiple of `num_gpus` num_groups: number of expert groups num_nodes: number of server nodes, where the intra-node network (e.g, NVLink) is faster num_gpus: number of GPUs, must be a multiple of `num_nodes` Returns: physical_to_logical_map: [layers, num_replicas], the expert index of each replica logical_to_physical_map: [layers, num_logical_experts, X], the replica indices for each expert expert_count: [layers, num_logical_experts], number of physical replicas for each logical expert rrrr) rrr r=rfullrr r:r2rr) rr>r r/r0rr;r+r<r,Znum_redundant_expertsZ maxlogcntZlog2phyr r rrebalance_expertss4           r@) __doc__rr6inttupler%r-r=r@__all__r r r rsN  0 ! P 7