o p i-@sddlmZddlZddlZddlmZmZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZer^dd lmZdd lmZdd lmZddlmZeejZ         d0d1d(d)Z d2d3d.d/Z!dS)4) annotationsN) TYPE_CHECKINGLiteral)GroupShardedOptimizerStage2)GroupShardedStage2)GroupShardedStage3)GroupShardedScaler)MixPrecisionOptimizer) get_logger) Optimizer)Sequence) GradScaler)Group)LayerFmodelr optimizerr levelLiteral['os', 'os_g', 'p_g_os']scalerGradScaler | Nonegroup Group | Noneoffloadbool sync_buffersbuffer_max_sizeint segment_size sync_commdp_group exclude_layerSequence[str | int] | Nonereturn#tuple[Layer, Optimizer, GradScaler]c  Csdtdd} | dvs| tjvsJdt|tjjs#Jdt|tt fs.Jd|dvs6Jdd d } t t | | }|d urRt |dkrRtd |d vr|tdtdtdt|j|||| | d}t|||||| | d}n|dkrt||||||| | | | d }ntdt|tjjrt|}tdtdtd|||fS)a Use group_sharded_parallel can perform group shared configuration on the model, optimizer and GradScaler. Level has three string options, 'os', 'os_g' and 'p_g_os' corresponds to three different usage scenarios: optimizer state segmentation, optimizer state + gradient segmentation, and parameter + gradient + optimizer state segmentation. Usually, optimizer state + gradient segmentation is actually a re optimization of optimizer state segmentation, so optimizer state + gradient segmentation can be used to realize optimizer state segmentation. Args: model (Layer): The layer to be wrapped with group_sharded_parallel. optimizer (Optimizer): The optimizer to be wrapped with group_sharded_parallel. level (str): The different level of the group sharded. Such as `os`, `os_g`, `p_g_os`. scaler (GradScaler|None, optional): If AMP is used, you need to pass GradScaler. Defaults to None, indicating that GradScaler is not used. group (Group|None, optional): The group instance. Defaults to None, indicating that the default environment group is used. offload (bool, optional): Whether to use the offload function. Defaults to False, which means that the offload function is not used. sync_buffers (bool, optional): Whether to broadcast model buffers. It is generally used when there are registered model buffers. Defaults to False, indicating that model buffers are not used. buffer_max_size (int, optional): The max size of the buffer used to integrate gradient in `os_g`. The larger the size, the more GPU memory will be used. Defaults to 2**23, which means that the dimension of the buffer is 2**23. segment_size (int, optional): The smallest size of parameter to be sharded in `p_g_os`. Defaults to 2**20, indicating that the dimension of the minimum segmented parameter is 2**20. sync_comm (bool, optional): Whether to use synchronous communication, only in `p_g_os` used. Defaults to False, indicating that asynchronous communication is used. dp_group(Group|None, optional): dp communication group, support to combine stage2 or stage3 with dp hybrid communication. exclude_layer(list|None, optional): exclude some layers for slicing for sharding stage3, for example, exclude_layer=["GroupNorm", id(model.gpt.linear)], exclude_layer must contain the layers' name or one layer's id. Returns: model: A wrapper for group sharded given model. optimizer: A wrapper for group sharded given optimizer. scaler: A wrapper for group sharded given scaler. Examples: .. code-block:: python >>> # type: ignore >>> # doctest: +REQUIRES(env:DISTRIBUTED) >>> import paddle >>> from paddle.nn import Linear >>> from paddle.distributed import fleet >>> from paddle.distributed.sharding import group_sharded_parallel >>> fleet.init(is_collective=True) >>> group = paddle.distributed.new_group([0, 1]) >>> model = Linear(1000, 1000) >>> clip = paddle.nn.ClipGradByGlobalNorm(clip_norm=1.0) >>> optimizer = paddle.optimizer.AdamW(learning_rate=0.001, parameters=model.parameters(), weight_decay=0.00001, grad_clip=clip) >>> # wrap sharding model, optimizer and scaler >>> model, optimizer, scaler = group_sharded_parallel(model, optimizer, "p_g", scaler=scaler) >>> img, label = data >>> label.stop_gradient = True >>> img.stop_gradient = True >>> out = model(img) >>> loss = paddle.nn.functional.cross_entropy(input=out, label=label) >>> loss.backward() >>> optimizer.step() >>> optimizer.clear_grad() :r)ZgpuZxpuzBgroup_sharded_parallel only support gpu, xpu and custom_device nowz2The model must be the instance of paddle.nn.Layer.zhThe optimizer must be the instance of paddle.optimizer.Optimizer or MixPrecisionOptimizer for main grad.)osos_gp_g_osz%The level must be os, os_g or p_g_os.cSs |jtjkSN)ZdtypepaddleZfloat16)paramr-p/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/paddle/distributed/sharding/group_sharded.py check_dtypes z+group_sharded_parallel..check_dtypeNzjthe input of scaler is None, please ensure the logic of your scaler outside is same as GroupShardedScaler.)r'r(z******************************z6Sharded level os uses sharded level os_g achieved now.)paramsZoptimrrr!device)rrrr!r1r)) rrrrrr r!r1r"zPlease enter the correct level.zIf there is a communication hang using group sharded, please check whether the communication operations of each process are unified.)r+Z get_devicesplitr1Zget_all_custom_device_type isinstancennrr r listfilter parameterslenlogger_warninginforZ_parameter_listrr ValueErrorampr r)rrrrrrrrrr r!r"r1r/Z params_fp16r-r-r.group_sharded_parallel2sF         r>outputstrOptimizer | NoneNonecCstdtj|rJd|dtj|ddtj|d}t|tr0t |j |n t|t rL|jr:dnd}|j|dt |j |ntd |d urmt|d s]Jd tj|d }t |j |tdd S)aV Group sharded encapsulated model and optimizer state saving module. Note: If using save_group_sharded_model saves the model. When loading again, you need to set the model or optimizer state before using group_sharded_parallel. Args: model (Layer): A wrapper for group sharded given model. output (str): Save directory. optimizer (Optimizer, optional): Group sharded encapsulated optimizer. Defaults to None, indicating that the optimizer state is not saved. Examples: .. code-block:: python >>> # type: ignore >>> # doctest: +REQUIRES(env:DISTRIBUTED) >>> import paddle >>> from paddle.nn import Linear >>> from paddle.distributed import fleet >>> from paddle.distributed.sharding import group_sharded_parallel, save_group_sharded_model >>> fleet.init(is_collective=True) >>> group = paddle.distributed.new_group([0, 1]) >>> model = Linear(1000, 1000) >>> clip = paddle.nn.ClipGradByGlobalNorm(clip_norm=1.0) >>> optimizer = paddle.optimizer.AdamW(learning_rate=0.001, parameters=model.parameters(), weight_decay=0.00001, grad_clip=clip) >>> # wrap sharding model, optimizer and scaler >>> model, optimizer, scaler = group_sharded_parallel(model, optimizer, "p_g", scaler=scaler) >>> img, label = data >>> label.stop_gradient = True >>> img.stop_gradient = True >>> out = model(img) >>> loss = paddle.nn.functional.cross_entropy(input=out, label=label) >>> loss.backward() >>> optimizer.step() >>> optimizer.clear_grad() >>> # save model and optimizer state_dict >>> save_group_sharded_model(model, optimizer, output=output_dir) zC==========Begin to save group sharded model and optimizer==========zSaving directory (z#) should be a directory, not a fileT)exist_okz model.pdmodelF) convert2cpuzBPlease use the layer which is wrapped with group_sharded_parallel.N_optimzFPlease use the optimizer which is wrapped with group_sharded_parallel.z model.pdoptzA==========End to save group sharded model and optimizer==========)r9r;r'pathisfilemakedirsjoinr3rr+saveZ_layerZ state_dictrZ_offloadZget_all_parametersr<hasattrrE)rr?rZ output_modelrDZ output_optr-r-r.save_group_sharded_models41     rL) NNFFrrFNN)rrrr rrrrrrrrrrrrrrr rr!rr"r#r$r%r*)rrr?r@rrAr$rB)" __future__rloggingr'typingrrr+ZNpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_optimizer_stage2rZDpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_stage2rZDpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_stage3rZCpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_utilsrZ2paddle.distributed.fleet.utils.mix_precision_utilsr Z"paddle.distributed.utils.log_utilsr Zpaddle.optimizerr collections.abcr Z paddle.ampr Z&paddle.distributed.communication.grouprZ paddle.nnrWARNINGr9r>rLr-r-r-r.s>