o p i"@s^ddlmZmZddlmZddlmZmZmZddZ ddZ dd d Z d d Z d dZ dS))_C_ops _legacy_C_ops)check_variable_and_dtype) LayerHelperin_dynamic_modein_dynamic_or_pir_modecCsXtr t||Sd}t|fit}|j|jd}|j|d|id|id|id|S)aO calculate the expert count according to the gate index. Args: numbers (Tensor): Tensor. The input gate index whose data type should be int32 or int64. upper_range (int): The number of the experts. Returns: out (Tensor): The output expert count. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> from paddle.distributed.models.moe import utils >>> numbers = [[0, 2], [0, 2]] >>> upper_range = 6 >>> numbers = paddle.to_tensor(numbers, dtype="int64") >>> number_count = utils._number_count(numbers, upper_range) >>> print(number_count) Tensor(shape=[6], dtype=int64, place=Place(gpu:0), stop_gradient=True, [2, 0, 2, 0, 0, 0]) number_countdtypenumbersOut upper_rangetypeinputsoutputsattrs)rrrrlocals"create_variable_for_type_inferencer append_op)r r op_typehelperoutrj/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/paddle/distributed/models/moe/utils.py _number_counts rcCshtr t|||dSd}t|fit}|j|jd}|j||g|g|dgdd|gid|S)a Assign pos decides which tokens should be fetched belong to specially expert orderly. Args: x (Tensor): Tensor. Every element in the list must be a Tensor whose data type should be float16, float32, float64, int32 or int64. cum_count (Tensor): The cumulative sum tokens of counters. Every element in the list must be a Tensor whose data type should be int64. Returns: out (Tensor): Assemble numbers in the order of counters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> from paddle.distributed.models.moe import utils >>> number_count = [2, 0, 2, 0] >>> numbers = [[0, 2], [0, 2]] >>> number_count = paddle.to_tensor(number_count, dtype="int64") >>> numbers = paddle.to_tensor(numbers, dtype="int64") >>> num_cum = paddle.cumsum(number_count) >>> pos = utils._assign_pos(x=numbers, cum_count=num_cum) >>> print(pos) Tensor(shape=[4], dtype=int64, place=Place(gpu:0), stop_gradient=True, [2, 0, 3, 1]) assign_posr )X cum_countZ eff_num_lenr )rrr)rrrrrrr r)xrrrrrrr _assign_pos?s r!cCs,|dkrtrt|||Stdtd)ab random routing topk gate idx ``` out = topk_idx for i in len(topk_idx): if topk * value[i][topk-1] < prob[i]: out[i][topk-1] = -1 ``` Args: topk_idx: gate idx, shape=(N, topk) topk_value: values, shape = topk_idx.shape prob: random prob, shape=(topk_idx.shape[0],) r"z$Not supporting static graph mode nowzonly topk=2 is supported now)rrZrandom_routing RuntimeError)Ztopk_idxZ topk_valueZprobZtopkrrr_random_routingqs r$cCs\tr t|||Sd}t|fit}|j|jd}|j|||dd|id|id|S)aq limit the expert count by capacity. Args: expert_count (Tensor): Tensor. The input expert count whose data type should be int32 or int64. capacity (Tensor): Tensor. The input capacity whose data type should be int32 or int64 and the elements of capacity should be the same with expert_count.numel()/n_work. n_work (int): The number of the works. Returns: out (Tensor): The output expert count limit by capacity. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> from paddle.distributed.models.moe import utils >>> expert_count = [1, 2, 2, 8, 3, 6] >>> capacity = [5, 5, 5] >>> n_work = 2 >>> expert_count = paddle.to_tensor(expert_count, dtype="int64") >>> capacity = paddle.to_tensor(capacity, dtype="int64") >>> out = utils._limit_by_capacity(expert_count, capacity, n_work) >>> print(out) Tensor(shape=[6], dtype=int64, place=Place(gpu:0), stop_gradient=True, [1, 2, 2, 4, 3, 3]) limit_by_capacityr ) expert_countcapacityr n_workerr)rrr%rrrr r)r&r'r(rrrrrr_limit_by_capacitysr)cCs~tr t||||St|dddgdt|dddgdtd it}|j|jd}|jd||dd |i||d d |S)a prune gate by capacity(only support CUDA) Args: gate_idx (Tensor): Represents the gate_id sequence corresponding to the input data with type int32, int64. expert_count (Tensor): The quantity value counted on the gate_id sequence of the input data with type int32, int64. n_worker(int, optional): The number of workers on the trainer with type int64. Returns: new_gate_idx (Tensor): The gate_id sequence corresponding to the new input data after passing through prune. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> from paddle.distributed.models.moe import utils >>> gate_idx = paddle.to_tensor([1, 3, 3, 3, 3, 2, 1, 1], dtype='int64') >>> expert_count = paddle.to_tensor([0, 3, 1, 3, 0, 0, 0, 0], dtype='int64') >>> n_worker = 1 >>> n_expert = 8 >>> new_gate_id = utils._prune_gate_by_capacity( ... gate_idx, expert_count, n_expert, n_worker ... ) >>> print(new_gate_id) Tensor(shape=[8], dtype=int64, place=Place(gpu:0), stop_gradient=True, [1, 3, 3, 3, -1, 2, 1, 1]) GateIdxZint32Zint64z/paddle.distributed.utils.prune_gate_by_capacity ExpertCountprune_gate_by_capacityr )r*r+Z NewGateIdx)n_expertr(rN)r,) rrr,rrrrr r)Zgate_idxr&r-r(rZ new_gate_idxrrr_prune_gate_by_capacitys6r.N)r")ZpaddlerrZpaddle.common_ops_importrZpaddle.frameworkrrrrr!r$r)r.rrrrs ' 2 ,