o 0 i5@sddlZddlmZmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlm?Z?ddl@mAZAddlBmCZCddlDmEZEddlFmGZGide%de)de+d e7d!e!d"e?d#eAd$e5d%e/d&e9d'e;d(e-d)e1d*eEd+e'd,eGd-eCe3e#e=d.ZHide de de d%ed ed!e d"ed#ed$ed'ed(ed)ed&ed*ed+e d,ed-eee ed.ZIeJeKZLGd/d0d0ZMGd1d2d2ZNd3eOfd4d5ZPd6eOfd7d8ZQd9d:ZRdS);N)OptionalUnion) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8Config FPQuantConfig GPTQConfig HiggsConfig HqqConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FPQuantHfQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerZawqZbitsandbytes_4bitZbitsandbytes_8bitZgptqZaqlmZquantoZquarkZfp_quantZeetqZhiggsZhqqzcompressed-tensorsZ fbgemm_fp8ZtorchaoZbitnetZvptqZspqr)Zfp8z auto-roundZmxfp4c@s.eZdZdZedefddZeddZdS)AutoQuantizationConfigz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictcCs|d}|dds|ddr!|ddrdnd}tj|}n|dur)td|tvr;td|d ttt|}||S) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsr3r4suffix target_clsrGh/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/transformers/quantizers/auto.pyrCys    z AutoQuantizationConfig.from_dictcKsVtj|fi|}t|dddurtd|d|j}||}|jdi||S)Nquantization_configz)Did not found a `quantization_config` in z2. Make sure that the model is correctly quantized.rG)rfrom_pretrainedgetattrr>rIrCupdate)rDpretrained_model_name_or_pathkwargsZ model_configr3rIrGrGrHrJs  z&AutoQuantizationConfig.from_pretrainedN)__name__ __module__ __qualname____doc__ classmethoddictrCrJrGrGrGrHr2ss r2c@sdeZdZdZedeeeffddZeddZ edeeefde efdd Z e d d Z d S) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. rIcKsxt|tr t|}|j}|tjkr|jr|d7}n|d7}|tvr0t d|dt t t|}||fi|S)Nr8r7r:r;) isinstancerTr2rCr4rr=r5rAr>r@rB)rDrIrNr4rFrGrGrH from_configs     zAutoHfQuantizer.from_configcKstj|fi|}||S)N)r2rJrW)rDrMrNrIrGrGrHrJs zAutoHfQuantizer.from_pretrainedquantization_config_from_argscCs|durd}nd}t|trt|trt|}nt|}|dur9|jj|jjkr9td|jjd|jjdt|tt tt t t frg|durg| }|D] \}}t|||qP|dt|d7}|dkrwt|t swt||St||S) z handles situations where both quantization_config from args and quantization_config from model config are present. NzYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)rVrTrrCr2 __class__rOr>rr rr rZget_loading_attributesitemssetattrr@rBwarningswarnloggerinfo)rDrIrXZ warning_msgZloading_attr_dictattrvalrGrGrHmerge_quantization_configss8       z*AutoHfQuantizer.merge_quantization_configscCs|dd}|dds|ddr"|ddrdnd}tj|}n|dur*td|tvr@td|d tt d dSd S) Nr4r5Fr6r7r8r9r:r;z~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) r<rr=r>r?r_warningr@rArB)r3r4rErGrGrHsupports_quant_methods   z%AutoHfQuantizer.supports_quant_methodN)rOrPrQrRrSrrrTrWrJrrc staticmethodrerGrGrGrHrUs   4rUmethodcfdd}|S)z-Register a custom quantization configuration.cs6tvr tddt|tstd|t<|S)NzConfig '' already registeredz*Config must extend QuantizationConfigMixin)r?r> issubclassr TypeErrorrDrgrGrHregister_config_fn   z8register_quantization_config..register_config_fnrG)rgrnrGrmrHregister_quantization_config   rpnamecrh)zRegister a custom quantizer.cs6tvr tddt|tstd|t<|S)Nz Quantizer 'riz!Quantizer must extend HfQuantizer)rAr>rjrrlrrrGrHregister_quantizer_fnroz1register_quantizer..register_quantizer_fnrG)rrrtrGrsrHregister_quantizerrqruc Cst|d}|rt|jsd}|s|dur,|r t|j||_n||_tj|j|d} nd} | durc| j|||||d| |}| |}| |}| |}t | jddsc| jj } t | d| |d<| |||fS)NrIF) pre_quantized)dtypefrom_tf from_flax device_map weights_onlyZ dequantizevalueZquant) hasattrrUrerIrcrWZvalidate_environmentZ update_dtypeZupdate_device_mapZupdate_tp_planZupdate_ep_planrKr4) configrIrwrxryrzr{ user_agentrvZ hf_quantizerr4rGrGrHget_hf_quantizer*s<       r)Sr]typingrrZmodels.auto.configuration_autorutilsrZutils.quantization_configrrr r r r r rrrrrrrrrrrrrrbaserZquantizer_aqlmrZquantizer_auto_roundrZ quantizer_awqr Zquantizer_bitnetr!Zquantizer_bnb_4bitr"Zquantizer_bnb_8bitr#Zquantizer_compressed_tensorsr$Zquantizer_eetqr%Zquantizer_fbgemm_fp8r&Zquantizer_finegrained_fp8r'Zquantizer_fp_quantr(Zquantizer_gptqr)Zquantizer_higgsr*Z quantizer_hqqr+Zquantizer_mxfp4r,Zquantizer_quantor-Zquantizer_quarkr.Zquantizer_spqrr/Zquantizer_torchaor0Zquantizer_vptqr1rAr?Z get_loggerrOr_r2rUstrrprurrGrGrGrHs  \                                )n