o * iG@sddlZddlZddlZddlZddlmZeddZdZdZddZ dd Z d d Z d d Z ddZ ddZddZddZddZddZddZdS)N) get_loggerINFOrootc Csdd}z5d}dg}t|dtj}d|}|D]}||dkr6|d}||}|WSq|WStyT}zt d |WYd}~dSd}~ww) z! get cpu info from lscpu cSsZg}|d}|D]!}t|ddt|dd}}|tt||dq |S)N,-rr)splitintextendlistrange)iZprocessed_cpu_infoZ cpu_rangesZ cpu_rangestartendrr/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/paddle/distributed/utils/process_utils.py_process_raw_cpu_info!s z,_get_cpu_info.._process_raw_cpu_infoNZlscpuutf-8noder_get_cpu_info failed, reason:) subprocess check_outputdecoderoslinesepfindstrip Exceptionloggerwarning) numa_idrZcpuscmdoutputZnuma_keylineZ raw_cpu_infoerrr _get_cpu_infos&  r&cC tdS)z* check if nvidia-smi is available nvidia-smishutilwhichrrrr_has_nvidia_smi< r,cCr')z' check if xpu-smi is available xpu-smir)rrrr _has_xpu_smiCr-r/cCs6t|dkr dS|d}t||krdS||S)Nrr)lenrr)Zstr_device_list local_rankZvisible_devicesrrr_get_xpu_device_from_envJs   r2cCs\td}|durt||Std}|durt||Std}|dur*t||St|S)z3 get currently used xpu physical device id ZXPULINK_VISIBLE_DEVICESNZXPU_VISIBLE_DEVICESCUDA_VISIBLE_DEVICES)rgetenvr2str)r1Zxpulink_visible_devicesZxpu_visible_devicescuda_visible_devicesrrr_get_xpu_deviceSs      r7cCsDtd}|dus |dkrt|S|d}t||krdS||S)z3 get currently used gpu physical device id r3Nr)rr4r5rr0)r1r6rrr_get_gpu_devicehs   r9c Cspzdddd|g}tj|ddd}|d}|WSty7}ztd |WYd }~d Sd }~ww) z+ get gpu numa info from nvidia-smi r(topoz-Cz-irtimeoutrrN)rrrrrrrr )Zgpu_idr"r#r!r%rrr_get_gpu_numa_infousr>cCst|}gd}tddkrgd}tj|ddd}||dd }g}|d D]}|d \}}|t t|t|dq0|S) N)r.r:z-mZCUDA_DEVICE_ORDERZOAM_ID)r.r:z-mor;rrrr) r rr4rrr splitlinesrr r )Zxpu_idr"r#Z cpu_affinity affinity_maskZaffinity_rangerrrrr_get_xpu_affinity_masksrCcCsts tdtStttddd}t|}|dur$tdtSt |}|dur3tdtS|dkr>td tSt |}|durMtd tSt d}t d |t d|t d}t d |tS) z set affinity for gpu zQnvidia-smi is not available, set_affinity is aborted, plz check your environment.PADDLE_LOCAL_RANK0rNzgFailed to get device id from cuda_visible_devices, set_affinity is aborted, plz check your environment.zMFailed to get numa info, set_affinity is aborted, plz check your environment.zN/Aznvidia-smi topo return numa id as N/A, set_affinity is aborted, plz check your environment. (Notice: This is expected behavior when executed on single numa node environment)zLFailed to get cpu info, set_affinity is aborted, plz check your environment.Check affinity before setting: zcheck affinity after setting: )r,rr FAIL_CODEmaxr rr4r9r>r&sched_getaffinityinfosched_setaffinity SUCCESS_CODE)r1 device_idr!rBaffinityrrrset_affinity_gpusF   rOcCsts tdtStttddd}t|}|dur$tdtSt |}t d}t d|t d|t d}t d|t S) z set affinity for xpu zNxpu-smi is not available, set_affinity is aborted, plz check your environment.rDrErNzMFailed to get device id, set_affinity is aborted, plz check your environment.rFzCheck affinity after setting: )r/rr rGrHr rr4r7rCrIrJrKrL)r1rMrBrNrrrset_affinity_xpus&   rPcCs.tjrtStjrtStdtS)Nz-Currently set_affinity only supports gpu env.) paddleZdeviceZis_compiled_with_cudarOZis_compiled_with_xpurPrr rGrrrr set_affinitys   rR)rr*rrQZ"paddle.distributed.utils.log_utilsrrrLrGr&r,r/r2r7r9r>rCrOrPrRrrrrs&     )