o ưi @sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlmZddlZddlmZddlmZddlmZddlTdd lmZmZd d lmZd d lmZd d lm Z d dl!m"Z"d dl#m$Z$d dl%m&Z&d dl'm(Z(d dl)m*Z*d dl+m,Z,d dl-m.Z.d dl/m0Z0ddZ1Gddde2eZ3GdddZ4e5j6dddgdfde e5de e2de e2d e e2d!e e e7f d"d#Z8e5j6dddgdfde e5de e2de e2d e e2d!e e e7f d$d%Z9d&d'Z:dS)(N)Enum)AnyDictListOptionalTupleUnion) BaseModel)verbose_logger)CACHED_STREAMING_CHUNK_DELAY)ModelParamHelper)*)EmbeddingResponseall_litellm_params)AzureBlobCache) BaseCache) DiskCache) DualCache)GCSCache) InMemoryCache)QdrantSemanticCache) RedisCache)RedisClusterCache)RedisSemanticCache)S3CachecCs:zt|tjrt|WdSWdStyYdSwN)r debuglitellmZ set_verboseprint Exception)Zprint_statementr!N/home/app/Keep/.python/lib/python3.10/site-packages/litellm/caching/caching.py print_verbose(s  r#c@seZdZdZdZdS) CacheMode default_on default_offN)__name__ __module__ __qualname__r%r&r!r!r!r"r$1sr$cR@sneZdZejejddddddddgddddddddddddddddddddddddddddddf(deedeedeedeed eed eed ee d ee d ee dee dee e deedeedeedeedeedee dee e efdeedeedeedeedeedeedeedeedeed ed!eed"eed#ee d$eed%eed&eed'eed(eed)ed*eed+eed,eefPd-d.Zd/efd0d1Zd2ed3ed/eefd4d5Zd3ed/efd6d7Zd8ed9eed/eefd:d;Zd3ed/efdd?Zd@ed/dfdAdBZedCed/efdDdEZdFed/efdGdHZdIdJZdKeedLee fdMdNZdrdOeefdPdQZ drdOeefdRdSZ dTdUZ!dVdWZ" drdOeefdXdYZ#dZed[eed/e$fd\d]Z% ^dsd_e&d`ed3edaed/e'eeeff dbdcZ( drdOeefdddeZ)dfdgZ*dhdiZ+djdkZ,dldmZ-dndoZ.d/e fdpdqZ/dS)tCacheN completionZ acompletion embeddingZ aembeddingZatranscriptionZ transcriptionZatext_completionZtext_completionZarerankZrerank responsesZ aresponsesTztext-embedding-ada-002typemodehostportpassword namespacettldefault_in_memory_ttldefault_in_redis_ttlsimilarity_thresholdsupported_call_typesazure_account_urlazure_blob_containers3_bucket_names3_region_names3_api_version s3_use_ssl s3_verifys3_endpoint_urls3_aws_access_key_ids3_aws_secret_access_keys3_aws_session_token s3_configs3_pathgcs_bucket_namegcs_path_service_accountgcs_path$redis_semantic_cache_embedding_modelredis_semantic_cache_index_nameredis_flush_sizeredis_startup_nodesdisk_cache_dirqdrant_api_baseqdrant_api_keyqdrant_collection_nameqdrant_quantization_config%qdrant_semantic_cache_embedding_model!qdrant_semantic_cache_vector_sizegcp_service_accountgcp_ssl_ca_certsc),Ksb|tjkrM|std}*|*durt|*trt|*}|r?|||||d|)}+|'dur.|'|+d<|(dur6|(|+d<tdi|+|_ nt d||||d|)|_ nq|tj krbt d|||| ||d|)|_ n\|tj krtt|!|"|#| |$|%|&d|_ nJ|tjkr~t|_ n@|tjkrtd|||||||||||d |)|_ n&|tjkrt|||d |_ n|tjkrt| | d |_ n |tjkrt| d |_ d tjvrtjd d tjvrtjd d tjvrtjd | |_ ||_!||_"||_#||_$|pt%j&|_'|j!tjkr|dur||_$|j!tjks|j!tj kr| dur| |_$|j"dur-t|j t r/|j"|j _"dSdSdS)a& Initializes the cache based on the given type. Args: type (str, optional): The type of cache to initialize. Can be "local", "redis", "redis-semantic", "qdrant-semantic", "s3" or "disk". Defaults to "local". # Redis Cache Args host (str, optional): The host address for the Redis cache. Required if type is "redis". port (int, optional): The port number for the Redis cache. Required if type is "redis". password (str, optional): The password for the Redis cache. Required if type is "redis". namespace (str, optional): The namespace for the Redis cache. Required if type is "redis". ttl (float, optional): The ttl for the Redis cache redis_flush_size (int, optional): The number of keys to flush at a time. Defaults to 1000. Only used if batch redis set caching is used. redis_startup_nodes (list, optional): The list of startup nodes for the Redis cache. Defaults to None. # Qdrant Cache Args qdrant_api_base (str, optional): The url for your qdrant cluster. Required if type is "qdrant-semantic". qdrant_api_key (str, optional): The api_key for the local or cloud qdrant cluster. qdrant_collection_name (str, optional): The name for your qdrant collection. Required if type is "qdrant-semantic". similarity_threshold (float, optional): The similarity threshold for semantic-caching, Required if type is "redis-semantic" or "qdrant-semantic". # Disk Cache Args disk_cache_dir (str, optional): The directory for the disk cache. Defaults to None. # S3 Cache Args s3_bucket_name (str, optional): The bucket name for the s3 cache. Defaults to None. s3_region_name (str, optional): The region name for the s3 cache. Defaults to None. s3_api_version (str, optional): The api version for the s3 cache. Defaults to None. s3_use_ssl (bool, optional): The use ssl for the s3 cache. Defaults to True. s3_verify (bool, optional): The verify for the s3 cache. Defaults to None. s3_endpoint_url (str, optional): The endpoint url for the s3 cache. Defaults to None. s3_aws_access_key_id (str, optional): The aws access key id for the s3 cache. Defaults to None. s3_aws_secret_access_key (str, optional): The aws secret access key for the s3 cache. Defaults to None. s3_aws_session_token (str, optional): The aws session token for the s3 cache. Defaults to None. s3_config (dict, optional): The config for the s3 cache. Defaults to None. # GCS Cache Args gcs_bucket_name (str, optional): The bucket name for the gcs cache. Defaults to None. gcs_path_service_account (str, optional): Path to the service account json. gcs_path (str, optional): Folder path inside the bucket to store cache files. # Common Cache Args supported_call_types (list, optional): List of call types to cache for. Defaults to cache == on for all call types. **kwargs: Additional keyword arguments for redis.Redis() cache Raises: ValueError: If an invalid cache type is provided. Returns: None. Cache is set as a litellm param ZREDIS_CLUSTER_NODESN)r1r2r3rLZ startup_nodesrUrV)r1r2r3rL)r1r2r3r8embedding_modelZ index_name)rOrPZcollection_namer8Zquantization_configrWZ vector_size) r<r=r>r?r@rArBrCrDrErF)Z bucket_nameZpath_service_accountrI)Z account_url container)rNcacher!)(LiteLLMCacheTypeZREDISrZ get_secret isinstancestrjsonloadsrrYrZREDIS_SEMANTICrZQDRANT_SEMANTICrLOCALrZS3rZGCSrZ AZURE_BLOBrZDISKrinput_callbackappendsuccess_callbacklogging_callback_manageradd_litellm_success_callback_async_success_callback"add_litellm_async_success_callbackr9r/r4rLr5r$r%r0),selfr/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVkwargsZ_env_cluster_nodesZcluster_kwargsr!r!r"__init__8s p                   zCache.__init__returnc Ksd}|jdi|}|durtd||St}t}|D]?}||vr<|||}|dur;|t|dt|7}q||vr]tj dur]||durLq||}|t|dt|7}qtd|t |}|j |fi|}|j dd|i||S) a  Get the cache key for the given arguments. Args: **kwargs: kwargs to litellm.completion() or embedding() Returns: str: The cache key generated from the arguments, or None if no cache key could be generated. Nz Returning preset cache key: %sz: Tz Created cache key: %spreset_cache_keyr!)!_get_preset_cache_key_from_kwargsr rr Z_get_all_llm_api_paramsr_get_param_valuer\rZ3enable_caching_on_provider_specific_optional_paramsr*_get_hashed_cache_key_add_namespace_to_cache_key_set_preset_cache_key_in_kwargs) rgrh cache_keyrlZcombined_kwargsZlitellm_param_kwargsparamZ param_valueZhashed_cache_keyr!r!r" get_cache_keys:       zCache.get_cache_keyrsrhcCs,|dkr ||S|dkr||S||S)z? Get the value for the given param from kwargs modelfile)_get_model_param_value_get_file_param_value)rgrsrhr!r!r"rn@s   zCache._get_param_valuecCs`|dipi}|dipi}|dipi}|dp!|d}|||}|p/|p/|dS)a Handles getting the value for the 'model' param from kwargs 1. If caching groups are set, then return the caching group as the model https://docs.litellm.ai/docs/routing#caching-across-model-groups 2. Else if a model_group is set, then return the model_group as the model. This is used for all requests sent through the litellm.Router() 3. Else use the `model` passed in kwargs metadatalitellm_params model_groupru)get_get_caching_group)rgrhryrzZmetadata_in_litellm_paramsr{Z caching_groupr!r!r"rwNs zCache._get_model_param_valueryr{cCs2|dg}|r|D] }||vrt|Sq dS)Ncaching_groups)r|r\)rgryr{r~groupr!r!r"r}_s  zCache._get_caching_groupcCsL|d}|di}|di}|dp%t|ddp%|dp%|dS)zo Handles getting the value for the 'file' param from kwargs. Used for `transcription` requests rvryrzZ file_checksumnameN file_name)r|getattr)rgrhrvryrzr!r!r"rxis     zCache._get_file_param_valuecKs |rd|vr|dddSdS)a  Get the preset cache key from kwargs["litellm_params"] We use _get_preset_cache_keys for two reasons 1. optional params like max_tokens, get transformed for bedrock -> max_new_tokens 2. avoid doing duplicate / repeated work rzrlN)r|)rgrhr!r!r"rmws z'Cache._get_preset_cache_key_from_kwargsrlcKs$|rd|vr||dd<dSdSdS)z Set the calculated cache key in kwargs This is used to avoid doing duplicate / repeated work Placed in kwargs["litellm_params"] rzrlNr!)rgrlrhr!r!r"rqs z%Cache._set_preset_cache_key_in_kwargsrrcCs&t|}|}td||S)z Get the hashed cache key for the given cache key. Use hashlib to create a sha256 hash of the cache key Args: cache_key (str): The cache key to hash. Returns: str: The hashed cache key. zHashed cache key (SHA-256): %s)hashlibsha256encode hexdigestr r)rrZ hash_objecthash_hexr!r!r"ros  zCache._get_hashed_cache_keyrcKsP|di}|dp|didp|j}|r |d|}td||S)a If a redis namespace is provided, add it to the cache key Args: hash_hex (str): The hashed cache key. **kwargs: Additional keyword arguments. Returns: str: The final hashed cache key with the redis namespace. rYr4ryZredis_namespace:zFinal hashed key: %s)r|r4r r)rgrrhZdynamic_cache_controlr4r!r!r"rps  z!Cache._add_namespace_to_cache_keyccsLd}tdt||D]}ddd||||digiVttq dS)NrchoicesdeltaZ assistant)Zrolecontent)rangelentimesleepr )rgr chunk_sizeir!r!r"generate_streaming_contents z Cache.generate_streaming_content cached_resultmax_agecCs|durHt|trHd|vrH|d}t}||}|dur#||kr#dS|d}zt|tr1W|St|}W|StyGt|}Y|Sw|S)zL Common get cache logic across sync + async implementations N timestampresponse) r[dictrr|r]r^r ast literal_eval)rgrrr current_timeZ response_agecached_responser!r!r"_get_cache_logics.    zCache._get_cache_logicdynamic_cache_objectcKsz[|jd i|durWdS|dg}d|vr|d}n|jd i|}|durY|di}|dp<|dps                   j  =  2