o yq i6@sdZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZd d lmZmZmZmZmZmZmZd d lmZd d lmZmZe r^d d l m!Z!e"e#Z$e%dZ&dZ'dddZ(eGdddZ)e    dBde e)de e*de*de*de e*de e*de e+e*e*fde e,e*de-e,e+e,e+e e*ffddZ.Gdd d eZ/Gd!d"d"eZ0   dCd#d$d%e+de e*de e+e*e*fde e*ddf d&d'Z1d(e+fd)d*Z2d+e+fd,d-Z3d.e+fd/d0Z4d#d$d1e*ddfd2d3Z5d#d$d4e+d5e6d1e*ddf d6d7Z7d4e+d8e)d5e6de,e*fd9d:Z8d;e,e+dZ9d#d$d?e,e*d5e6de,e+fd@dAZ:dS)Dz.Git LFS related type definitions and utilitiesN) dataclass)ceil)getsize) TYPE_CHECKINGBinaryIOIterableOptional TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_url get_sessionhf_raise_for_status http_backoffloggingvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-Typec@s`eZdZUdZeed<eed<eed<edefddZ edefd d Z ed e fd d Z dS) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathcCs\t|}t|d}|ddd}t|}Wdn1s"wY||||dS)Nrbrrr)rioopenpeekr)clsrrfilershar&Z/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/huggingface_hub/lfs.py from_pathJs  zUploadInfo.from_pathdatacCs&t|}|t||dd|dS)Nr)rrr)rdigestlen)r#r)r%r&r&r' from_bytesRs zUploadInfo.from_bytesfileobjcCsD|d}|dtjt|}|}|dtj||||dS)Nrrr)readseekr SEEK_SETrtell)r#r-rr%rr&r&r' from_fileobjWs zUploadInfo.from_fileobjN) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr(r,rr2r&r&r&r'r7s  r upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturncCs|dur|ntj}d}|tjvrtj|}|d||d} d|dur&|nddgdd |Dd d } |dur?d t|i| d <itt|d|pJi}tj| || d} t| | } | dd} t | t slt d| d}t |trx|nd}dd | Ddd | D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HfHubHTTPError`] If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcSsg|] }|j|jdqS)oidr)rhexr).0rGr&r&r' s z'post_lfs_batch_info..r) operationrCobjectsZ hash_algonameref)r=rBjsonrPzMalformed response from serverZtransfercSsg|] }d|vrt|qSerror)_validate_batch_actionsrMobjr&r&r'rNcSsg|] }d|vrt|qSrU)_validate_batch_errorrXr&r&r'rNrZ)r ZENDPOINTZREPO_TYPES_URL_PREFIXESr LFS_HEADERSr rpostrrTget isinstancelist ValueErrorr;)r<r=r>r?r@rArBrCZ url_prefixZ batch_urlpayloadrespZ batch_inforPZchosen_transferr&r&r'post_lfs_batch_infoasB,        rdc@seZdZUeed<eed<dS) PayloadPartT partNumberetagN)r3r4r5r9r8r;r&r&r&r'res  rec@s&eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rKpartsN)r3r4r5r6r;r8r`rer&r&r&r'rhs rhrOrlfs_batch_actionc CsHt||d}|durtd|jddS|dd}t||dd}|dur2t||di}|d} t|d |d } | durjzt| } Wntt fy`td | d wt ||| | d nt || d|durt|t|d |} t j | t||d|jj|jjdd} t| t|jddS)a9 Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HfHubHTTPError`] If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrGverifyheader chunk_sizehref)rAzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rOrmrn upload_url)rOrp)r=rBrJrSz: Upload successful)rWr^loggerdebugZ path_in_repo_validate_lfs_actionrr9ra TypeError_upload_multi_part_upload_single_partrr]r upload_inforrLrr) rOrjr=rBrArk upload_action verify_actionrmrnrpZ verify_urlZ verify_respr&r&r' lfs_uploadsB        rz lfs_actioncCs:t|dtr|ddust|dtstd|S).validates response from the LFS batch endpointrormNz"lfs_action is improperly formatted)r_r^r;dictra)r{r&r&r'rs s rslfs_batch_actionscCspt|dtrt|dtstd|did}|did}|dur.t||dur6t||S)r|rKrz)lfs_batch_actions is improperly formattedrkrGrlN)r_r^r;r9rars)r~rxryr&r&r'rWs rWlfs_batch_errorcCsht|dtrt|dtstd|d}t|tr.t|dtr.t|dts2td|S)r|rKrz'lfs_batch_error is improperly formattedrVmessagecode)r_r^r;r9rar})r error_infor&r&r'r["s  r[rpcCsH|jdd}td||d}t|WddS1swYdS)aK Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Raises: [`HfHubHTTPError`] If the upload resulted in an error. TZ with_tqdmPUTr)N)as_filerr)rOrpr-responser&r&r'rv0s "rvrmrncCsJt||j|d}t|||d}tj|t||jjtd}t |dS)z@ Uploads file using HF multipart LFS transfer protocol. )rmrwrn)rOsorted_parts_urlsrn)rTrBN) _get_sorted_parts_urlsrw_upload_parts_iterativelyrr]_get_completion_payloadrrLr\r)rOrmrnrprresponse_headersZcompletion_resr&r&r'ruDs rurwcCsNddtdd|DdddD}t|}|t|j|kr%td|S)NcSsg|]\}}|qSr&r&)rM_rpr&r&r'rNZsz*_get_sorted_parts_urls..cSs2g|]\}}|rt|dkrt|d|fqS)r )isdigitr+r9)rMZpart_numrpr&r&r'rN]s  cSs|dS)Nrr&)tr&r&r'bsz(_get_sorted_parts_urls..)keyz0Invalid server response to upload large LFS file)sorteditemsr+rrra)rmrwrnZsorted_part_upload_urlsZ num_partsr&r&r'rYs rrrKcCsfg}t|D]'\}}|d}|dus|dkr#td|d|d||d|dq||dS)NrgrEzInvalid etag (`z`) returned for part r )rfrg)rKri) enumerater^raappend)rrKriZ part_numberrmrgr&r&r'rks  rrc Csg}|jdd<}t|D].\}}t||||d}td||d}t|||jWdn1s6wYq Wd|S1sGwY|S)NTr)Z seek_fromZ read_limitrr)rrrrrrrB) rOrrnrBr-Zpart_idxZpart_upload_urlZ fileobj_sliceZpart_upload_resr&r&r'rzs&   r)NNNN)NNN);r6r re dataclassesrmathrZos.pathrtypingrrrrr urllib.parser Zhuggingface_hubr utilsr rrrrrrZ utils._lfsrZ utils.sharrZ _commit_apirZ get_loggerr3rqcompileZ OID_REGEXZLFS_MULTIPART_UPLOAD_COMMANDr\rr;r}r`tuplerdrerhrzrsrWr[rvr9rurrrr&r&r&r's     $    )  V  F