o rq i2a%@sddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z mZmZddlmZmZmZmZmZmZddlmZddlmZdd lmZd d lmZmZd d l m!Z!d d l"m#Z#d dl$m%Z%m&Z&m'Z'd dl(m)Z)d dl*m+Z+m,Z,m-Z-m.Z.eZ/dddddddddddddeddfde0de e0dee0e dfde ee e0fde e1de ede ee0e e0fde ee0e e0fde e0de ee e0e0fde ee e0e0fde2de0d e e0d!e e1d"e ee!d#e0f"d$d%Z3edddddddddddf d&e0de e0dee0e dfde e0de ee e0fde e1de ede ee0e e0fde ee0e e0fde ee e0e0fde ee e0e0fd!e e1de2d#e0fd'd(Z4dedddddddddddd) de0d e e0de e0dee0e dfde ee e0fde e1de ede ee0e e0fde ee0e e0fde e0de ee e0e0fde ee e0e0fde2d"e ee!fd*d+Z5d,d-Z6d.e0fd/d0Z7d1ee0e e0ffd2d3Z8d1e e0fd4d5Z9deddddddddf d6e e0d7e)d8e0de0d9ed:e0d;e0d e e0de e0de ede ee0e e0fde ee0e e0fde ee e0e0fde ee e0e0fde2dZ:dS)?N) nullcontext) CookieJarPath)DictListOptionalTypeUnion)DEFAULT_DATASET_REVISIONDEFAULT_MODEL_REVISIONINTRA_CLOUD_ACCELERATIONREPO_TYPE_DATASETREPO_TYPE_MODELREPO_TYPE_SUPPORT)get_modelscope_cache_dir) get_logger)thread_executor)HubApiModelScopeConfig)ProgressCallback)InvalidParameter)$create_temporary_directory_and_cache download_fileget_file_download_url)ModelFileSystemCache)get_model_masked_directorymodel_id_to_group_owner_name strtoboolweak_file_lockFmodel_idrevision cache_dir user_agentlocal_files_onlycookiesignore_file_patternallow_file_pattern local_dirallow_patternsignore_patterns max_workersrepo_id repo_typeenable_file_lockprogress_callbacksreturncCs| p|} | s td| tvrtd| dt|dur$| tkr"tnt}|dur1ttjdd}|rY|dur9|nt }tj tj |ddd tj |d| d d }t|}nt}|t| | ||||||||| | | |d WdS1s{wYdS) a Download all files of a repo. Downloads a whole snapshot of a repo's files at the specified revision. This is useful when you want all files from a repo, because you don't know which ones you will need a priori. All files are nested inside a folder in order to keep their actual filename relative to that folder. An alternative would be to just clone a repo but this would require that the user always has git and git-lfs installed, and properly configured. Args: repo_id (str): A user or an organization name and a repo name separated by a `/`. model_id (str): A user or an organization name and a model name separated by a `/`. if `repo_id` is provided, `model_id` will be ignored. repo_type (str, optional): The type of the repo, either 'model' or 'dataset'. revision (str, optional): An optional Git revision id which can be a branch name, a tag, or a commit hash. NOTE: currently only branch and tag name is supported cache_dir (str, Path, optional): Path to the folder where cached files are stored, model will be save as cache_dir/model_id/THE_MODEL_FILES. user_agent (str, dict, optional): The user-agent info in the form of a dictionary or a string. local_files_only (bool, optional): If `True`, avoid downloading the file and return the path to the local cached file if it exists. cookies (CookieJar, optional): The cookie of the request, default None. ignore_file_pattern (`str` or `List`, *optional*, default to `None`): Any file pattern to be ignored in downloading, like exact file names or file extensions. allow_file_pattern (`str` or `List`, *optional*, default to `None`): Any file pattern to be downloading, like exact file names or file extensions. local_dir (str, optional): Specific local directory path to which the file will be downloaded. allow_patterns (`str` or `List`, *optional*, default to `None`): If provided, only files matching at least one pattern are downloaded, priority over allow_file_pattern. For hugging-face compatibility. ignore_patterns (`str` or `List`, *optional*, default to `None`): If provided, files matching any of the patterns are not downloaded, priority over ignore_file_pattern. For hugging-face compatibility. max_workers (`int`): The maximum number of workers to download files, default 8. enable_file_lock (`bool`): Enable file lock, this is useful in multiprocessing downloading, default `True`. If you find something wrong with file lock and have a problem modifying your code, change `MODELSCOPE_HUB_FILE_LOCK` env to `false`. progress_callbacks (`List[Type[ProgressCallback]]`, **optional**, default to `None`): progress callbacks to track the download progress. Raises: ValueError: the value details. Returns: str: Local folder path (string) of repo snapshot Note: Raises the following errors: - [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) if `use_auth_token=True` and the token cannot be found. - [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if ETag cannot be determined. - [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) if some parameter value is invalid z*Please provide a valid model_id or repo_idInvalid repo type: z, only support: NMODELSCOPE_HUB_FILE_LOCKtrue.lockTexist_ok/___) r/r#r$r%r&r'r(r)r*r,r+r-r1) ValueErrorrrr r rosenvirongetrmakedirspathjoinreplacer r_snapshot_download)r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1 system_cache lock_filecontextrGg/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/modelscope/hub/snapshot_download.pysnapshot_download!sNI    $rI dataset_idc Cs| dur ttjdd} | r5|dur|nt} tjtj| dddtj| d|dd}t |}nt }|t |t ||||||||| | | d WdS1sVwYdS) aZ Download raw files of a dataset. Downloads all files at the specified revision. This is useful when you want all files from a dataset, because you don't know which ones you will need a priori. All files are nested inside a folder in order to keep their actual filename relative to that folder. An alternative would be to just clone a dataset but this would require that the user always has git and git-lfs installed, and properly configured. Args: dataset_id (str): A user or an organization name and a dataset name separated by a `/`. revision (str, optional): An optional Git revision id which can be a branch name, a tag, or a commit hash. NOTE: currently only branch and tag name is supported cache_dir (str, Path, optional): Path to the folder where cached files are stored, dataset will be save as cache_dir/dataset_id/THE_DATASET_FILES. local_dir (str, optional): Specific local directory path to which the file will be downloaded. user_agent (str, dict, optional): The user-agent info in the form of a dictionary or a string. local_files_only (bool, optional): If `True`, avoid downloading the file and return the path to the local cached file if it exists. cookies (CookieJar, optional): The cookie of the request, default None. ignore_file_pattern (`str` or `List`, *optional*, default to `None`): Any file pattern to be ignored in downloading, like exact file names or file extensions. Use regression is deprecated. allow_file_pattern (`str` or `List`, *optional*, default to `None`): Any file pattern to be downloading, like exact file names or file extensions. allow_patterns (`str` or `List`, *optional*, default to `None`): If provided, only files matching at least one pattern are downloaded, priority over allow_file_pattern. For hugging-face compatibility. ignore_patterns (`str` or `List`, *optional*, default to `None`): If provided, files matching any of the patterns are not downloaded, priority over ignore_file_pattern. For hugging-face compatibility. enable_file_lock (`bool`): Enable file lock, this is useful in multiprocessing downloading, default `True`. If you find something wrong with file lock and have a problem modifying your code, change `MODELSCOPE_HUB_FILE_LOCK` env to `false`. max_workers (`int`): The maximum number of workers to download files, default 8. Raises: ValueError: the value details. Returns: str: Local folder path (string) of repo snapshot Note: Raises the following errors: - [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) if `use_auth_token=True` and the token cannot be found. - [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if ETag cannot be determined. - [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) if some parameter value is invalid Nr4r5r6Tr7r9r:) r/r#r$r%r&r'r(r)r*r,r+r-) rr<r=r>rr?r@rArBr rrCr)rJr#r$r*r%r&r'r(r)r+r,r0r-rDrErFrGrGrHdataset_snapshot_downloads:A    $rK) r/r#r$r%r&r'r(r)r*r+r,r-r1c Cs|st}|tvrtd|tft|| ||d\}}|dur |nt}|r;t|jdkr0tdt d|| St j |dt td}tdkrdtd pUt}|rdtd |||d <t}|j||d }|durvt }|tkrJ| rtj| }n|rtjj|g|d R}ntjj|dg|d R}td|d||j||||d}|d}dtjvri|ddi}ni|ddi}|jdur|j|d<|j ||d|durdn|||d}t!|||||dd||||||| | | || dd|vrIt"||}tj#|rtdntd|dztj$tj||dd Wnt%yHt d!|d"tj|dYnswnq|t&kr| rYtj| }n|rjtjj|g|d R}ntjj|d#g|d R}td$|t'|\}}|pt(}td%t)||||}|durt*d&|dSt!|||||||||||||| | | || d|j+|d'| }|S)(Nz'Invalid repo type: %s, only support: %s)r*r$r/rzCannot find the requested files in the cached path and outgoing traffic has been disabled. To enable look-ups and downloads online, set 'local_files_only' to False.z6We can not confirm the cached file is for revision: %s)r%)z user-agentzsnapshot-identifierr5ZINTRA_CLOUD_ACCELERATION_REGIONz6Intra-cloud acceleration enabled for downloading from zx-aliyun-region-id)r.r/r9modelszDownloading Model from z to directory: )r#r'endpointZRevisionZCI_TESTzsnapshot-ci-testTrueZSnapshotcached_model_revisionTF)r"r# recursiveZ use_cookiesheadersrM) r/r#r'r(r)r,r+r-rMr1.z3Target directory already exists, skipping creation.zCreating symbolic link [z].)target_is_directoryzFailed to create symbolic link z for Zdatasetsz"Downloading Dataset to directory: z"Fetching dataset repo file list...z*Failed to retrieve file list for dataset: )Z revision_info),rrrrrlen cached_filesr;loggerwarningZget_root_locationrZget_user_agentstruuiduuid4r r<getenvrZ!_get_internal_acceleration_domaininfoZget_endpoint_for_readZ get_cookiesr@abspathrAsplitprintZget_valid_revision_detailr=rOZget_model_files_download_file_listsrexistssymlinkOSErrorrrr fetch_repo_fileserrorZsave_model_version)r.r/r#r$r%r&r'r(r)r*r+r,r-r1temporary_cache_dircacherDrQZ region_id_apirM directoryZrevision_detailZsnapshot_header repo_filesZmasked_directorygroup_or_ownernameZcache_root_pathrGrGrHrCs0                  rCc Csd}d}g} z|j||dd|||d}Wnty0}ztd|WYd}~|Sd}~ww||t||kr? |S|d7}q)NrTr9)r.r# root_pathrP page_number page_sizerMzError fetching dataset files: )Zget_dataset_files ExceptionrVreextendrT) rhr.r#rMrorprjZ dataset_fileserGrGrHrds4     rdpatterncCs&zt|WdStyYdSw)NTF)recompile BaseException)rtrGrGrH_is_valid_regexs   rxpatternscCs*t|tr|g}|durdd|D}|S)NcSs"g|] }|ds |n|dqS)r9*)endswith).0itemrGrGrH sz'_normalize_patterns..) isinstancerX)ryrGrGrH_normalize_patternss rcCs0|durg}|D] }t|r||q|SdS)N)rxappend)ryZregex_patternsr}rGrGrH_get_valid_regex_patterns rrjrgrfapirlrkrMc st|}t| } t| } t| } t| }g}|D]ddkr!qzm|r1tfdd|Dr1Wq| r@tfdd| Dr@Wq|rOtfdd|DrOWq| durb| rbtfdd| DsbWq| duru| rutfd d| DsuWqrtjd }td |d WqWnt y}z t d |WYd}~qd}~ww| qt |dd f dd}t |dkrtdt |d||td d ddSdS)Nr treecg|] }td|qSrfnmatchr|rt repo_filerGrHr~z(_download_file_lists..crrrrrrGrHr~rcs g|] }t|dduqS)NameN)rusearchrrrGrHr~ sc3 |] }td|VqdSrNrrrrGrH   z'_download_file_lists..c3rrrrrrGrHrrrzFile z8 already in cache with identical hash, skip downloading!z The file pattern is invalid : %sF)r- disable_tqdmc sp tkrt|d d}n tkr j|d d}n td dtt|| dddS)Nr)r" file_pathr#rM) file_nameZ dataset_name namespacer#rMr3z, supported types: F)rr1)rrrZget_dataset_file_urlrrr)rurl) rrgr'rMrkrQrlr1r.r/r#rfrGrH_download_single_file(s8 z3_download_file_lists.._download_single_filerzGot z files, start to download ...z Download z 'z' successfully.)rranyrar<r@basenamerVdebugrqrWrrrTr\)rjrgrfr.rrlrkrQr/r#r'r(r)r+r,r-rMr1Zignore_regex_patternZfiltered_repo_filesrrsrrG) rrgr'rMrkrQrlr1rr.r/r#rfrHr`sj         $ r`);rr<rurY contextlibrhttp.cookiejarrpathlibrtypingrrrr r Zmodelscope.utils.constantr r r rrrZmodelscope.utils.file_utilsrZmodelscope.utils.loggerrZmodelscope.utils.thread_utilsrrrrcallbackrerrorsrZ file_downloadrrrZ utils.cachingrZ utils.utilsrrrr rVrXboolintrIrKrCrdrxrrr`rGrGrGrHs                  v       b        1