o
rq i!2 @ s d dl Z d dlZd dlZd dlZd dlZd dlmZmZ d dlm Z d dl
m Z m Z d dl
mZ d dlmZ e Zee d dkZ G d d
d
eZG d d d eZdS )
N)movermtree)Dict) FILE_HASH)MODELSCOPE_ENABLE_DEFAULT_HASH_VALIDATION) compute_hash)
get_loggerFalsetruec @ st e Zd ZdZdZdZdZ defddZdd Z d
d Z
d d
Z dd Z dd Z
dd Zdd Zdd Zdd ZdS )FileSystemCachez.mscz.mdlidz.mvcache_root_locationc K s* t | _tj|dd || _| dS )zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN) threadingRLock _cache_lockosmakedirsr
load_cache)selfr
kwargs r c/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/modelscope/hub/utils/caching.py__init__ s
zFileSystemCache.__init__c C s | j S N)r
r r r r get_root_location1 s z!FileSystemCache.get_root_locationc C sd g | _ tj| jtj}tj|r0t|d}t
|| _ W d d S 1 s)w Y d S d S )Nrb) cached_filesr pathjoinr
r
KEY_FILE_NAMEexistsopenpickleload)r cache_keys_file_pathfr r r r 4 s
"zFileSystemCache.load_cachec C s | j e tj| jtj}tjd| jd\}}z$t |d}t
| j | W d n1 s0w Y t
|| W n$ ty_ zt| W n tyR Y nw tj|r^t| w W d dS 1 skw Y dS )a
Save cache metadata in order to verify that the cached content is consistent with the remote content.
Example of the cached content:
[{'Path': 'configuration.json', 'Revision': 'f01dxxx'}, {'Path': 'model.bin', 'Revision': '1159xxx'}, ...]
z.tmp)suffixdirwbN)r r r r! r
r r" tempfilemkstempfdopenr% dumpr r ExceptioncloseOSErrorr# unlink)r r' fdZ
temp_filenamer( r r r save_cached_files< s0
"z!FileSystemCache.save_cached_filesc C dS )zCheck the key is in the cache, if exist, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
Nr r keyr r r get_fileV s zFileSystemCache.get_filec C r6 )zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
Nr )r r8 locationr r r put_filea s
zFileSystemCache.put_filec C s& || j v r| j | | dS dS )zvRemove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r remover5 r7 r r r
remove_keym s
zFileSystemCache.remove_keyc C s | j D ] }||kr dS qdS )NTF)r )r r8 Z
cache_filer r r r# w s
zFileSystemCache.existsc C s t | j | dS )zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r
r r r r r clear_cache~ s
zFileSystemCache.clear_cachec C s t | S r )hashlibsha256encode hexdigestr7 r r r hash_name s zFileSystemCache.hash_nameN)__name__
__module__ __qualname__r" MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r r r5 r9 r; r= r# r> rC r r r r r s$
r c s e Zd ZdZd fdd Zdd Zdd Zd efd
d Zd d
Z dd Z
dd Z dd Z dd Z
dd Zdd Zdd Zd d Z ZS ) ModelFileSystemCachezLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
Nc sp |du s|du rt tj| | nt tj||| tjd||f i| _| |
| _ dS )a\ Put file to the cache
Args:
cache_root(`str`): The modelscope local cache root(default: ~/.cache/modelscope/hub)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
Nz%s/%s) superr r r r! load_model_metar rH
model_metasave_model_metaload_model_versionZcached_model_revision)r Z
cache_rootownername __class__r r r s
z ModelFileSystemCache.__init__c C sj t j| jtj}t j|r-t|d}t || _
W d d S 1 s&w Y d S tj di| _
d S )Nr unknown) r r r! r
r rG r# r$ r% r& rN rH r Zmeta_file_pathr( r r r rM s
"z$ModelFileSystemCache.load_model_metac C s\ t j| jtj}t j|r,t|d}| W d S 1 s%w Y d S d S )Nr)
r r r! r
r rI r# r$ readstrip)r model_version_file_pathr( r r r rP s
$z'ModelFileSystemCache.load_model_version
revision_infoc C s t j| jtj}t|d+}t|tr$d|d |d f }| | n
| | W d d S W d d S 1 s s r