o )i. @sddlZddlZddlmZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl mZer4ddlmZgdZdZded e d fd d Zd ed efddZd:ded efddZde jjd ede jjfddZded efddZdeded efddZd;de ed e efd!d"ZGd#d$d$Z d.decorator..wrapped)__name__r)rTrZrRrS)rTr" decorators zdeprecated..decorator)rr)rRrSr]r!r\r"rsrc@s:eZdZdZedeeeffddZddZddZ d S) rz A mixin for enums to allow aliasing of enum members Example: >>> class MyClass(Aliasable, int, Enum): >>> ... rcCstN)NotImplementedErrorr!r!r!r" get_aliasesszAliasable.get_aliasescCslt||jr |}|j|jkp||j|j||j|jkS|}||j|j}|||}||kSr^)rE __class__r`valuer )selfotheraliasesZ self_valueZ other_valuer!r!r"__eq__s   zAliasable.__eq__cCs|j|j|j}t|Sr^)rer rbhash)rcZcanonical_valuer!r!r"__hash__szAliasable.__hash__N) r[ __module__ __qualname____doc__ staticmethodrstrr`rfrhr!r!r!r"rs  rr9 shard_sizesr.cCsTt|||kr tdg}d}|D]}||}||||}|||}q|S)a Shards a tensor into a list of tensors along a given dimension. raises: ValueError: If the sum of shard_sizes does not match the size of the tensor along the given dimension. :param tensor: The input tensor to shard. :param shard_sizes : List of sizes for each shard along the specified dimension. :param dim : The dimension along which to shard the tensor. :returns: A list of tensors sharded along the specified dimension. zSSum of shard_sizes must equal the size of the tensor along the specified dimension.r)r4sizer8narrowappend)r9rnr.shardsZ start_idxroZend_idxshardr!r!r"rs rcs|stddd|D}t|dkrtdt|dj}tfdd|D|<tj||dj|djd }d}|D]}|j}| || |||7}q=|S) z Combine decompressed shards along a given dimension using `narrow`. :param shards: List of decompressed shard tensors. :param dim: Dimension to combine along (default: 0). :return: Combined decompressed tensor. zThe list of shards is empty.cSsh|]}|jqSr!)dtype.0rsr!r!r" sz!combine_shards..r,z$All shards must have the same dtype.rc3s|]}|jVqdSr^)shaperur-r!r" sz!combine_shards..)rtdevice) r8r?listrxr4r5ZzerosrtrzrpZcopy_)rrr.Z shard_dtypesZ total_shapecombinedZ shard_offsetrsZ shard_sizer!r-r"rs   r bytemaskscCs"tj|ddd}t|}|S)a Converts a bytemask tensor to a bitmask tensor to reduce memory. Shape RxC will be compressed to R x ceil(C/8) :param bytemasks: mask tensor where each byte corresponds to a weight :return: mask tensor where each bit corresounds to a weight r+little)axisbitorder)numpyZpackbitsr5 from_numpy)r}Zpacked_bits_numpyZpacked_bits_torchr!r!r"r(s rpacked_bitmasksoriginal_shapecCs8tj|d|ddd}t||t}|S)a# Converts a bitmask tensor back to a bytemask tensor for use during decompression :param packed_bitmasks: mask tensor where each bit corresponds to a weight :param original_shape: dense shape to decompress to :return: boolean mask of weights in the original dense shape r+r~)rcountr)rZ unpackbitscpur5rZreshapeZastypebool)rrZ unpacked_bitsZunpacked_bitmasks_torchr!r!r"r6s rbaseattrrbc csrt}t|||}t|||zdVW||ur!t|||dSt||dS||ur3t|||wt||w)a Patch the value of an object attribute. Original value is restored upon exit :param base: object which has the attribute to patch :param attr: name of the the attribute to patch :param value: used to replace original value Usage: >>> from types import SimpleNamespace >>> obj = SimpleNamespace() >>> with patch_attr(obj, "attribute", "value"): ... assert obj.attribute == "value" >>> assert not hasattr(obj, "attribute") N)objectrLr@delattr)rrrb _sentineloriginal_valuer!r!r"rPs   rc@s0eZdZdZdeegeffddZddZdS)ra  Similar to `collections.DefaultDict`, but upon fetching a key which is missing, the key is passed as arguments to the `default_factory` :param default_factory: function which takes a key as input and returns the corresponding default value default_factorycCs ||_dSr^)r)rcrr!r!r"__init__vs z!ParameterizedDefaultDict.__init__cCs,t|tr |j|}n||}|||<|Sr^)rEr/r)rckeyrbr!r!r" __missing__ys   z$ParameterizedDefaultDict.__missing__N)r[rirjrkrrrrr!r!r!r"rms r)r()NN)r)* contextlibrW functoolsrtypingrrrrrrrr5Z transformersr rr __all__r'rmr r rrnnModulerrrrrZTensorr1rrrrcontextmanagerrrdictrr!r!r!r"sZ         !