o )iX]@sdZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z mZmZddlZddlmZzddlmZmZmZmZmZddlmZmZmZmZd ZWneyod ZdZdZdZdZdZdZdZdZdZYnwgd Z d efd dZ! e!d ddej"j#de$fddZ%dej"j#dej&fddZ'e!dddej"j#de(dej)fddZ*dej"j#dej)de(fddZ+ dej"j#dej&fddZ, dIdej"j#d e(d!ej"j-d"e eej&e d#ffd$d%Z. dIdej"j#d e(dej)d"e eej&e d#ffd&d'Z/dej"j#d e(fd(d)Z0e!e1dej2dej"j#fd*d+Z3e!dd dId,eee efde(d-ej)d"e eej&e d#ffd.d/Z4e!ddd,eee efde(fd0d1Z5e!e1dej2dej"j#fd2d3Z6e!e1dej2 dId4eej"j#e ej"j#fd5e ej&fd6d7Z7d8ej"j#d e(dej"j#fd9d:Z8d8ej"j#d e(fd;d<Z9e!d=de&d>fdej"j#d5ej&d"eej&e d#fdej"j#fd?d@Z:dej"j#dej"j#fdAdBZ;ej2dCdDZ< e!d ddej"j#de$fdEdFZ=e!e1dej2 dIdej"j#d5e ej&fdGdHZ>dS)Jaw Utilities associated with offloading functionality provided by `accelerate`. | ------------------------------------------------------------------------------------------------------ | # noqa: E501 | Operation | Without offloading support | With offloading support | # noqa: E501 | ---------- | -------------------------------------- | ------------------------------------------------ | # noqa: E501 | Add | module.register_parameter(name, param) | register_offload_parameter(module, name, param) | # noqa: E501 | Check | N/A | has_offloaded_params(module) | # noqa: E501 | Onload | N/A | with align_module_device(module) | # noqa: E501 | Update | module.name.data.copy_(new_data) | update_offload_parameter(module, name, new_data) | # noqa: E501 | Delete | del module.name | delete_offload_parameter(module, name) | # noqa: E501 | Add Module | module.register_module(name, child) | register_offload_module(name, child) | # noqa: E501 | Del Module | del module.name | delete_offload_module(module, name) | # noqa: E501 | ------------------------------------------------------------------------------------------------------ | # noqa: E501 N)wraps) attrgetter)AnyCallableDictIterableLiteralOptionalTupleUnion) patch_attr)AlignDevicesHookadd_hook_to_moduleattach_align_device_hooknamed_module_tensorsremove_hook_from_module)OffloadedWeightsLoaderPrefixedDatasetfind_tied_parametersset_module_tensor_to_deviceTF)is_module_offloadedget_execution_deviceget_offloaded_deviceupdate_prefix_dictupdate_parameter_dataregister_offload_parameterupdate_offload_parameterdelete_offload_parameterhas_offloaded_paramsdisable_hf_hookdisable_offload align_modulesalign_module_deviceregister_offload_moduledelete_offload_moduleoffloaded_dispatchdisable_offloadingremove_dispatchfallbackcs dttgtfffdd }|S)Nfunccs<tsdkrt|dd}|St|fdd}|S|S)Nerrorc_std)Nz9Please install `accelerate` in order to use this function) ValueErrorargskwargsr/l/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/compressed_tensors/utils/offload.py fallback_fnasz8check_accelerate..decorator..fallback_fncsSNr/r,r(r/r0r1is)_has_accelerater)r)r1r3r/r0 decorator]s  z#check_accelerate..decorator)rr)r(r5r/r3r0check_accelerate\sr6r3modulereturncCst|Sr2)rr7r/r/r0rwsrcCs>t|rt|jjd}|jjj}||jSt|jS)zk :param module: module to check :return: device module is offloaded to onto after forward pass r) rlist_hf_hook weights_mapkeysdatasetdevicenext parameters)r7Z first_keyZprefix_datasetr/r/r0r|s   rkeydatacCs(t|std|jj}t|||dS)a Updates the offloaded state dict for a given module. Parameter named key is replaced by data. This is neccesary because parameter updates for offloaded modules do not persist automatically between loads. This function only affects the offloaded state dict and not the current state of the loaded module. :param module: module containing the parameter to update :param key: name of parameter to update :param data: tensor to update parameter with in the offloaded state dict z3Prefix dict is only applicable to offloaded modulesN)rr+r;r<offload_to_weights_map)r7rBrCr<r/r/r0rs rnew_param_data param_namecCst|||dS)aG Update the data of an existing parameter and its offload dict. Supports both parameters of offloaded modules and non-offloaded modules :param module: module containing the parameter to update :param new_param_data: tensor to update parameter with :param param_name: name of module parameter to update N)r)r7rErFr/r/r0rs rcCsb|D]}t|r|jjSt|jddd}|dur"|jSqtd|dt dS)z Get the device which inputs should be moved to before module execution. Assume that modules execute in the same order as returned by `model.modules()` :param module: module to check, may be offloaded :return: onload device of module FrecurseNz"Unable to get execution device of z, falling back to CPUcpu) modulesrr;execution_devicer@rAr?warningswarntorch)r7 submoduleparamr/r/r0rs    rname parameteroffload_devicediskcCstdd|D}|||t|rI|j}|jdusJ|j|j|<t|j||j ||j|}|j dur?i|j | <|sKt ||ddSdSdS)aj Register a parameter to the given module which may be offloaded :param module: maybe offloaded module :param name: name of newly registered parameter :param parameter: parameter being registered :param offload_device: device on which weight will be offloaded to. If None is provided, then infer device from parameters on module css |] }|jtdkVqdS)metaN)r?rN).0pr/r/r0 sz-register_offload_parameter..NrU) anyrAZregister_parameterrr;r<r?Zoriginal_devicesrDrCtied_params_mapdata_ptrr)r7rQrRrSZ has_onloadhook offloadedr/r/r0rs    rcCst||}|jj|jkrtd|jjd|j|jtdkr-||jur-|j|t|r>|j j }t ||||dSdS)a Update the data of an existing parameter and its offload dict. Supports both parameters of offloaded modules and non-offloaded modules :param module: module containing the parameter to update :param name: name of module parameter to update :param data: tensor to update parameter with :param offload_device: device on which weight will be offloaded to. If None is provided, then infer device from parameters on module z!Shape of parameter being updated z% does not match shape of update data rUN) getattrrCshaperLrMr?rNZcopy_rr;r<rD)r7rQrCrSrPr<r/r/r0rs   rcCs,t||t|r|jj}t||dSdS)z Delete a parameter from a module which may be offloaded :param module: maybe offloaded module :param name: name of parameter being deleted N)delattrrr;r<delete_from_weights_map)r7rQr<r/r/r0r s rc#sBifdd}||dVD] \}}t||qdS)Ncs$t|dr|j|<t|dSdS)Nr;)hasattrr;rr9hooksr/r0 collect_hookss   z&disable_hf_hook..collect_hooks)applyitemsr)r7rerOr\r/rcr0rs   rr<valuecCs*t|tr%|dkrtdt||j}|j|}t||||dSt|trN||jvr5|j |t |j dkrJ|dkrJt|j |||dSt dt|tr|dkr`tdt||dur||vrn||j}ntt|d}|durtd|j}|j|d||<dSt dt|) a Helper function which implements offloaded item assignment for PrefixedDataset, OffloadedWeightsLoader, and Dict types. :param weights_map: weight map to be updated with offload information :param key: key used to identify weight location :param value: weight being offloaded :param offload_device: device on which weight will be offloaded to. If None is provided, then infer device from parameters in weights_map rTz!Cannot offload to disk with type rz@Updating weights_map with disk offloading is not implemented yetNz2Cannot infer offload device from empty weights_mapr?>Updating offload data not implemented for weights_map of type ) isinstancerr+typer>prefixrDrall_keysappendlenindex state_dictNotImplementedErrordictr?r@itervaluesto)r<rBrhrSr>Ztensr/r/r0rD*s@      rDcCst|tr|j}|j|}t||dSt|tr.t|jdkr*t|j|dSt dt|t r8||=dSt dt |)NrzCDelete from weights_map with disk offloading is not implemented yetrj) rkrr>rmrarrprqrrrsrtrl)r<rBr>r/r/r0rahs"    raccs.t|rd|j_dVd|j_dSdVdS)z Context manager to disable module onloading and offloading. Parameters will stay on their current device :param module: module to disable offloading for FNT)rr;offloadr9r/r/r0r s    r rJrKccsvt|tjjr |fn|}t }|D]}|t|||t|qdVWddS1s4wYdS)a2 Context manager for onloading modules to a device, and disabling onload and offload attempts triggered by forward calls. Used for sequential onloading of layers :param modules: `torch.nn.Module` or iterable of `torch.nn.Module`s to onload :param execution_device: device to onload to N) rkrNnnModule contextlib ExitStack enter_contextr"r )rJrKstackr7r/r/r0r!s "r!basec Cst|rz|j}|js J|jdusJd}d}t|j}t|}t|||dD]*\}} | |} |j dur>i|j | <t |j|d|| |j rSt|||q)|j szt|jj|jj|dd} t|j|jd| ||d|j d} t|| |||dS)z Register a submodule with offloading if the parent module is offloaded :param base: module to attach submodule to :param name: name of submodule :param module: submodule to attach NFT)Zinclude_buffersrH.)rm)rKrxZio_same_devicer<offload_buffersplace_submodules skip_keysrZ)rr;rxr<r@rAr?rrrwrZr[rDrrrr>rmr rKrZregister_module) rrQr7r\rrZcurrent_devicerSrFrPr]r<Zsubmodule_hookr/r/r0r#sD      r#cCs8t||}t|D] \}}t||q t||dS)z Delete a submodule from a model which may contain offloading :param base: parent module to delete submodule from :param name: name of submodule on parent N)r^r:named_parametersrr`)rrQr7rF_r/r/r0r$s  r$r*rIc sdkrtdt||}fdd|D}t|d}t|}i}|D]}|D]}t||} i|| <q-q)t||d||dt |dd d d |S) a Unlike `dispatch_model`, this function forces a module (and its submodules) to offload all parameters and replace them with meta tensors, utiliizing the `AlignDevicesHook` to control onloading and offloading. :param module: module containing parameters to offload :param execution_device: device that modules will onload and execute on :param offload_device: device that module parameters will offload to :return: module with offloading device hooks rTz*Disk offloading is not currently supportedcsi|] \}}||qSr/)rw)rVrBvalrSr/r0 sz&offloaded_dispatch..)rrr?T)rKrxr<rZ hf_device_maprIzcuda:0)Z fake_offloadZ fake_exec) rsr'rrrgrrrr[rsetattr) r7rKrSrrr<Z tied_paramsrZgrouprFr[r/rr0r%s,  r%cCs$t|ddt|drt|d|S)z Remove any existing dispatches from module :param module: module which may be dispatched with hf hooks :return: module without dispatch TrGr)rrbr`r9r/r/r0r'.s   r'c#stjtdtffdd }ttd| dVWdn1s%wYD]!\}\}}||_|jddD] \}}t|||jq=| |dq.dS)z Keep modules onloaded and disable offloading until this context exits. Affects modules which have been hooked with accelerate's `AlignDevicesHook` selfcs8||g|Ri|}|vr||jf|<d|_|S)NF)rx)rr7r-r.retZonloaded_modulesZoriginal_pre_forwardr/r0keep_onload_pre_forwardFs z3disable_offloading..keep_onload_pre_forward pre_forwardNFrG) r rrtr rgrxrrrC post_forward)rr7r\rxrQrPr/rr0r&<sr&cCst|dot|jto|jjS)ad Checks if a module has offloaded parameters by checking if the given module has a AlignDevicesHook attached with offloading enabled Args: module (`torch.nn.Module`): The module to check for an offload hook. Returns: bool: `True` if the module has an offload hook and offloading is enabled, `False` otherwise. r;)rbrkr;r rxr9r/r/r0r^s  rc cst|r?|dur|jj}||j_z|j|dVW|j|d|dur-||j_dSdS|j|d|dur>||j_w|durdd|jddD}z |D]}t|||qQdVW|D] \}}t|||qbdS|D] \}}t|||qswdVdS)a~ Context manager that moves a module's parameters to the specified execution device. Args: module (`torch.nn.Module`): Module with parameters to align. execution_device (`torch.device`, *optional*): If provided, overrides the module's execution device within the context. Otherwise, use hook execution device or pass NcSsi|]\}}||jqSr/ri)rVrQrPr/r/r0rs z'align_module_device..FrG)rr;rKrrrrrg)r7rKZoriginal_deviceZdevicesrQr?r/r/r0r"ss:     r"r2)?__doc__r{rL functoolsroperatorrtypingrrrrrr r r rNZcompressed_tensors.utilsr Zaccelerate.hooksr rrrrZaccelerate.utilsrrrrr4 ImportError__all__r6ryrzboolrr?rstrZTensorrrr Parameterrrr nullcontextcontextmanagerrrDrar r!r#r$r%r'r&rr"r/r/r/r0s   (      , !   =   9 :