o ưi@shddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZddlZddlZddlZddlmZddlm Z ddl!m"Z"ddl#Z#ddl$Z#ddl%Z#ddl#m&Z&dd l'm(Z(dd l)m*Z*dd l+m,Z,m-Z-m.Z.m/Z/dd l0m1Z1dd l2m3Z3ddl4m5Z5ddl6m7Z7m8Z8ddl9m:Z:ddl;mZ>ddl?m@ZAddlBmCZCddlDmEZEddlFmGZGddlHmIZIddlJmKZKddlLmMZMddlNmOZOddlPmQZQddlRmSZSddlTmUZUddlVmWZWmXZXddlYmZZZm[Z[m\Z\dd l]m^Z^dd!l_m`Z`maZadd"lbmcZcmdZddd#lemfZfdd$lgmhZhmiZimjZjmkZkmlZldd%lmmnZnmoZompZpdd&lqmrZsdd'ltmuZumvZvdd(lwmxZxdd)lymzZzdd*l{m|Z|dd+l}m~Z~mZdd,lmZmZdd-lmZmZmZmZdd.lmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZdd/lmZdd0lmZmZmZdd1lmZdd1lmZdd2lmZmZmZdd3lmZmZmZmZmZmZmZmZmZmZd4d5lmZerdd6lmZdd7lmZmZdd8lmZeeefZneZeZeZeZGd9d:d:ejZGd;d<d<ZdS)=N defaultdict) lru_cache) TYPE_CHECKINGAnyAsyncGeneratorCallableDict GeneratorListLiteralOptionalTupleUnioncast) AsyncOpenAI) BaseModel)overload)get_secret_str)verbose_router_logger)uuid) DualCache InMemoryCache RedisCacheRedisClusterCache)DEFAULT_MAX_LRU_CACHE_SIZE) CustomLogger)run_async_function)!_get_parent_otel_span_from_kwargs&get_metadata_variable_name_from_kwargs)coroutine_checker)CredentialAccessor)tracer)Logging)SensitiveDataMasker)JSONProviderRegistry)RouterBudgetLimiting)LeastBusyLoggingHandler)LowestCostLoggingHandler)LowestLatencyLoggingHandler)LowestTPMLoggingHandler)LowestTPMLoggingHandler_v2simple_shuffle)get_deployments_for_tag) add_fallback_headers_to_responseadd_retry_headers_to_response)"_get_router_metadata_variable_namereplace_model_in_jsonlshould_replace_model_in_jsonl)InitalizeCachedClient)get_dynamic_litellm_paramsis_clientside_credential)filter_team_based_modelsfilter_web_search_deployments) CooldownCache)DEFAULT_COOLDOWN_TIME_SECONDS_async_get_cooldown_deployments/_async_get_cooldown_deployments_with_debug_info_get_cooldown_deployments_set_cooldown_deployments)#_check_non_standard_fallback_formatget_fallback_model_grouprun_async_fallback)!get_num_retries_from_retry_policy)#async_raise_no_deployment_exceptionsend_llm_exception_alert)DeploymentAffinityCheck)ModelRateLimitingCheck)PromptCachingDeploymentCheck)0increment_deployment_failures_for_current_minute1increment_deployment_successes_for_current_minute)FlowItem Scheduler)AllMessageValues FileTypesOpenAIFileObjectOpenAIFilesPurpose)#CONFIGURABLE_CLIENTSIDE_AUTH_PARAMSVALID_LITELLM_ENVIRONMENTSAlertingConfigAllowedFailsPolicyAssistantsTypedDictCredentialLiteLLMParamsCustomRoutingStrategyBase DeploymentDeploymentTypedDictGuardrailTypedDictLiteLLM_ParamsMockRouterTestingParamsModelGroupInfoOptionalPreCallChecks RetryPolicyRouterCacheEnumRouterGeneralSettingsRouterModelGroupAliasItemRouterRateLimitErrorRouterRateLimitErrorBasicRoutingStrategySearchToolTypedDict) ServiceTypes)CustomPricingLiteLLMParamsGenericBudgetConfigType LiteLLMBatch) ModelInfo)ModelResponseStreamStandardLoggingPayloadUsage) CustomStreamWrapperEmbeddingResponse ModelResponseRulesfunction_setupget_llm_provider!get_non_default_completion_params get_secretget_utc_datetimeis_region_allowed)PatternMatchRouter)Span) AutoRouterPreRoutingHookResponseComplexityRouterc@seZdZdZdS) RoutingArgs<N)__name__ __module__ __qualname__ttlrrE/home/app/Keep/.python/lib/python3.10/site-packages/litellm/router.pyrsrc_@seZdZUeZeed<dZeeed<dZ e ed<dZ dZ ee ed<dZeeed<dZeeeeeefed <ddddddddddidddddddddddd dgggiddd d diddddd diddeddf.deeeeeeeeffdeedeeedeeedeedeedee deedee deededeeede deedee dee dee deedeed eed!ee d"ed#e d$d%eeed&ed'ed(ed)eeeeee!ffd*ed+ed,ed-e d.eee"efd/eee"fd0ee d1ee#d2eed3eed4e d5d6ee$d7ed8ee%d9ee&d:eed;e dd?Z'e(d=eefd@dAZ)dBdCZ*dDdEZ+e(dFeeefd=ee,e-ffdGdHZ.dIe,fdJdKZ/d4ee0efd7efdLdMZ1dNdOZ2dPdQZ3dRdSZ4dTdUZ5dVdWZ6dXdYZ7dZd[Z8d\d]Z9d^d_Z:d`daZ;dbdcZd6ee$fdidjZ?dkefdldmZ@dnedoeeeefd=eeAeBffdpdqZCdnedoeeeefd=eeAeBffdrdsZDd=efdtduZEdvedoeefdwdxZFeGdnedoeeHdye d d=eBfdzd{ZIeG d dnedoeeHdye dd=eAfd|d{ZIeG d dnedoeeHdyee d e dfd=eeBeAffd}d{ZI d dnedoeeHdyefd~d{ZIe(deJdeeKd=dfddZLdeBdoeeeefded=eBfddZMdeBdoeeeefded=eBfddZNdvedoeefddZOdnedoeeeefd=eeAeBffddZP d dnededeed=dfddZQdeRdked=dfddZS d dedeed=dfddZT d dkededeed=eUfddZVe(dkeded=dfddZW d dkededeed=dfddZXdkedefddZYdeded=eeee ffddZZdeded=eeee ffddZ[deded=eeee ffddZ\deedoeeeeefeeeeefffddZ]dnedoeeeHfddZ^eGdnedoeeeefdye d d=eBfddZ_eG d dnedoeeeefdye dd=eAfddZ_ d dnedoeeeefdyefddZ_eG d dnedoeeHde dye dd=eAf ddZ`eGdnedoeeHde dye d d=eBf ddZ` d dnedoeeHde fddZ`dnede dedeaedfdeeeff ddZbdned=efddZcdnedoeeHdeeeffddZddednefddZedednefdd„ZfdednefddĄZgdednefddƄZhdeidnefddɄZjdeidnefdd˄ZkdnededefddτZldnefddфZmdnefddӄZn   d dnededeedeedeef dd؄Zo   d dnededeedeedeef ddڄZpdnedefdd܄Zq   d dednedeedeedeef dd߄ZrdednefddZsdefddZtdnedefddZudedefddZvdnedefddZwded=dfddZxdnedefddZydeeefdneded=eeeffddZzdnedefddZ{dnedefddZ| d dnedeeefdeed=e}fddZ~deeefdnefddZ d dnedeeefdeed=e}fddZdeeefdnefddZdned=efddZdned=efddZdneedffddZddZdned=efdd Zdned=efd d Z d dneed=efd d Zdned=efddZdned=efddZdnefddZ d dedeefddZ ddede dfddZ d dedeefddZ d dedeefddZdefd d!Z d dedeefd"d#Z  ddedeed$eefd%d&Zd'eRd(eed&eed'eed(eed)eededefd*d+Zed,d-Z    dded)eed&eed'eed(eef d.d/Zed0d1Zdefd2d3Z d ded)eefd4d5Z     dd6eRd7eed8eed'eed(eed9eef d:d;Zd<d=Z d d&eeeeefd)eed=eeefd>d?Zd=eefd@dAZ  dd'eRdBe de d7eed8eed=ee eff dCdDZdEdFZd=eefdGdHZd=efdIdJZdKeefdLdMZded=e dNfdOdPZded'eRd=efdQdRZdSedTeed=e fdUdVZd=efdWdXZdnedYeAded=efdZd[ZdnedTeefd\d]ZdnedTeed=eaeeeeffd^d_Zdkefd`daZ d dkedTeedbeefdcddZ  ddned7eedoeeeHdTeedeeedbeef dfdgZd)edhefdidjZdkedledmedned=eeUf dodpZdhed=efdqdrZdkeUfdsdtZdhed=efdudvZdkeUfdwdxZdkeUd=efdydzZdefd{d|ZdkeUd=eUfd}d~ZdkeUdednefddZdkeUd=eeUfddZdede d=dfddZ d dnedeed=dfddZdkeUd=eeUfddZded=eeUfddZÐded=eeUfddZĐded=eefddZŐded=eeUfddZƐded=eeeeffddZeG d dkeedfdeddd=efddZeGdkddeded=efddZ d dkeeedfdedeed=efddZɐded=eefddZʐded=eefddZːdeded=eefddZ͐d)eded=eefddZϐd)ed=eefddZАd)ed=eaee ee ffddZeeӐdd)ed=eefddZԐd)ed=eee ffddZ d dYed)eed=efddZded=dfddZdefddZ ddeeded=eefddZِded=efddZڐdeed=eefddZېdeded=eefddZ d dednedeed=efdd„Z  ddedeedeed=eefdĐdńZސd deed=eefdƐdDŽZ d dkedeed=eefdȐdɄZdeed=efdːd̄Z d deed=eefd͐d΄Z ddeedeed=eeefdϐdЄZddѐd҄ZddӐdԄZ   ddeedeedeed=eeeeffd֐dׄZded=efdؐdلZdڐdۄZdܐd݄Zd dސd߄Z d dned7edoeeeefdeeefddZdned=eefddZdned=efddZ    ddnedoeeeeefdeeeefdeedeeed=eaeeeefff ddZ    ddnedeedoeeeeefdeeeefdeedTeed=eeeeffddZ   ddnedeedoeeeeefdeeeefdeef ddZ   ddnedeedoeeeeefdeeeefdeef ddZ   ddnedeedoeeeeefdeeeefdeed=eef ddZ    ddnedoeeeeefdeeeefdeedeeef ddZ    ddnedoeeeeefdeeeefdeedeeef ddZd7eedeed=eefddZd7eed=eefddZ d dTeefddZ d deRd)eefddZdeRfddZddZdefddZddZddZdS(Router model_namesFcache_responsesidefault_cache_time_secondsNleastbusy_loggerlowesttpm_loggeroptional_callbacksINFOTrsimple-shuffle model_listassistants_config search_toolsguardrail_list redis_url redis_host redis_portredis_passwordredis_db cache_kwargscaching_groups client_ttlpolling_intervaldefault_priority num_retries max_fallbackstimeoutstream_timeoutdefault_litellm_paramsdefault_max_parallel_requests set_verbose debug_level)DEBUGrdefault_fallbacks fallbackscontext_window_fallbackscontent_policy_fallbacksmodel_group_aliasenable_pre_call_checksenable_tag_filteringtag_filtering_match_any retry_after retry_policymodel_group_retry_policy allowed_failsallowed_fails_policy cooldown_timedisable_cooldownsrouting_strategy)r least-busyusage-based-routinglatency-based-routingcost-based-routingusage-based-routing-v2optional_pre_call_checksrouting_strategy_argsprovider_budget_configalerting_configrouter_general_settingsdeployment_affinity_ttl_secondsignore_invalid_deploymentsreturnc/6Cs"||_|.|_||_||_||_||_ddlm}/|/|_dt _ |jdur9|dkr/t t jn |dkr9t t j|,p=t|_||_|pEg|_|pJg|_g|_i|_d}0d}1i}2| |_|dusg|dur|durd}0|durq||2d <|dury||2d <|durt||2d <|dur||2d <| durt d t| |2d<|2| ||2}1| rt jdurt jd"d|0i|2t _| |_t|1t d|_t!||1d|_"||_#d|_$||_%g|_&t'|_(i|_)i|_*i|_+|pi|_,i|_-i|_.|dur|/||j0|_1|D]}3d|3dvrd|j|3dd<qng|_0d|_2|#dur|#|_3nt j3|_3|%p't4|_5t6|j|j5d|_7|&|_8t |_9|durB||_:nt j:durMt j:|_:nt;j<|_:|durZ||_=nt j=duret j=|_=nt j>|_=|pnt j?|_@||_A| |_B|'|_C|p~t jD}4|jE|4d|4|_D|dust jFdur|pt jF}4|jDdur|jDGd|4ind|4ig|_D|pt jH|_H|pt jI}5|jE|5d|5|_ItJtK|_LtJtK|_MtJtK|_Ng|_O|pi}t jP||d|_Q||_R|jRSd||jRSdd|jRSdid| ii|_T |jU|'|)dd|_VtWt jXtYrt jZ[|j\nt jZ[|j\tWt j]tYr3t jZ^|j_n|j_gt _]tWt j`tYrGt jZa|jbn|jbgt _`tWt jctYr[t jZd|jen|jegt _c|)|_f|*|_g|-|_hd|_itjjk||jgdr|(dur|(Gdndg}(d|_l|!durtW|!tmrtnd"i|!|_ln tW|!tnr|!|_l|jldurt odp|jljqdd |"|_rd|_s|$durtW|$tmrttd"i|$|_sn tW|$ttr|$|_s|jsdurt od!p|jsjqdd |+|_u|(dur|v|(|judur|w|x|y|zdS)#a Initialize the Router class with the given parameters for caching, reliability, and routing strategy. Args: model_list (Optional[list]): List of models to be used. Defaults to None. redis_url (Optional[str]): URL of the Redis server. Defaults to None. redis_host (Optional[str]): Hostname of the Redis server. Defaults to None. redis_port (Optional[int]): Port of the Redis server. Defaults to None. redis_password (Optional[str]): Password of the Redis server. Defaults to None. cache_responses (Optional[bool]): Flag to enable caching of responses. Defaults to False. cache_kwargs (dict): Additional kwargs to pass to RedisCache. Defaults to {}. caching_groups (Optional[List[tuple]]): List of model groups for caching across model groups. Defaults to None. client_ttl (int): Time-to-live for cached clients in seconds. Defaults to 3600. polling_interval: (Optional[float]): frequency of polling queue. Only for '.scheduler_acompletion()'. Default is 3ms. default_priority: (Optional[int]): the default priority for a request. Only for '.scheduler_acompletion()'. Default is None. num_retries (Optional[int]): Number of retries for failed requests. Defaults to 2. timeout (Optional[float]): Timeout for requests. Defaults to None. default_litellm_params (dict): Default parameters for Router.chat.completion.create. Defaults to {}. set_verbose (bool): Flag to set verbose mode. Defaults to False. debug_level (Literal["DEBUG", "INFO"]): Debug level for logging. Defaults to "INFO". fallbacks (List): List of fallback options. Defaults to []. context_window_fallbacks (List): List of context window fallback options. Defaults to []. enable_pre_call_checks (boolean): Filter out deployments which are outside context window limits for a given prompt model_group_alias (Optional[dict]): Alias for model groups. Defaults to {}. retry_after (int): Minimum time to wait before retrying a failed request. Defaults to 0. allowed_fails (Optional[int]): Number of allowed fails before adding to cooldown. Defaults to None. cooldown_time (float): Time to cooldown a deployment after failure in seconds. Defaults to 1. routing_strategy (Literal["simple-shuffle", "least-busy", "usage-based-routing", "latency-based-routing", "cost-based-routing"]): Routing strategy. Defaults to "simple-shuffle". routing_strategy_args (dict): Additional args for latency-based routing. Defaults to {}. alerting_config (AlertingConfig): Slack alerting configuration. Defaults to None. provider_budget_config (ProviderBudgetConfig): Provider budget configuration. Use this to set llm_provider budget limits. example $100/day to OpenAI, $100/day to Azure, etc. Defaults to None. deployment_affinity_ttl_seconds (int): TTL for user-key -> deployment affinity mapping. Defaults to 3600. ignore_invalid_deployments (bool): Ignores invalid deployments, and continues with other deployments. Default is to raise an error. Returns: Router: An instance of the litellm.Router class. Example Usage: ```python from litellm import Router model_list = [ { "model_name": "azure-gpt-3.5-turbo", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/", "api_key": , "api_version": , "api_base": }, }, { "model_name": "azure-gpt-3.5-turbo", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/", "api_key": , "api_version": , "api_base": }, }, { "model_name": "openai-gpt-3.5-turbo", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo", "api_key": , }, ] router = Router(model_list=model_list, fallbacks=[{"azure-gpt-3.5-turbo": "openai-gpt-3.5-turbo"}]) ``` r)ServiceLoggingTrrlocalNZredisurlhostportpasswordzwDeprecated 'redis_db' argument used. Please remove 'redis_db' from your config/database and use 'cache_kwargs' instead.dbtype) redis_cacheZin_memory_cache)rrmodellitellm_params)cacheZdefault_cooldown_time)fallback_param*)paramsZ router_objr max_retriesmetadatarrr)rrrouter_budget_limitingz+Router Custom Retry Policy Set: {}Z exclude_nonez3Router Custom Allowed Fails Policy Set: {}r){rrrrrrZlitellm._service_loggerrservice_logger_objlitellmZsuppress_debug_inforsetLevelloggingrrr`rrrrdeployment_namesZdeployment_latency_maprstrwarningupdate_create_redis_cacherCacherrrrK schedulerrdefault_deploymentrprovider_default_deployment_idsrypattern_routerteam_pattern_routers auto_routerscomplexity_routersr model_id_to_deployment_index_map model_name_to_deployment_indicesset_model_listrhealthy_deployments_access_groups_cacherr:rr9cooldown_cacherZ failed_callsropenaiZDEFAULT_MAX_RETRIESrZROUTER_MAX_FALLBACKSrequest_timeoutrrrrrvalidate_fallbacksrappendrrrint total_calls fail_calls success_callsprevious_modelsZChatchatr setdefaultZdeployment_statsrouting_strategy_init access_groups isinstance_async_success_callbacklistlogging_callback_managerZ"add_litellm_async_success_callbackdeployment_callback_on_successsuccess_callbackadd_litellm_success_callback#sync_deployment_callback_on_success_async_failure_callbackZ"add_litellm_async_failure_callback$async_deployment_callback_on_failurefailure_callbackZadd_litellm_failure_callbackdeployment_callback_on_failurerrrZrouter_budget_loggerr&Z!should_init_router_budget_limiterrdictr^infoformat model_dumprrrSradd_optional_pre_call_checks_initialize_alertinginitialize_assistants_endpointinitialize_router_endpointsapply_default_settings)6selfrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrZ cache_typer cache_configmZ _fallbacksZ_content_policy_fallbacksrrr__init__s                                               zRouter.__init__cCs.ttj}|j|j}d|vr|d|S)zS Returns a list of valid arguments for the Router.__init__ method. r)inspectgetfullargspecrrargs kwonlyargsremove)Zarg_specZ valid_argsrrrget_valid_argss   zRouter.get_valid_argscCsg}||dS)z; Apply the default settings to the router. N)r )rZdefault_pre_call_checksrrrrs zRouter.apply_default_settingscCstjtj|tjtj|tjtj|tjtj|tjtj|tjtj|tjtj ||j durM|j D]}tjjtj |ddq@dSdS)z Pseudo-destructor to be invoked to clean up global data structures when router is no longer used. For now, unhook router's callbacks from all lists NF)Z require_self) rrZ#remove_callback_from_list_by_objectrrrrinput_callbackZservice_callback callbacksr)rcallbackrrrdiscards8  zRouter.discardrcCs^|d}|std}|durt|trt|}|r(tdii|d|iStdi|S)za Initializes either a RedisCache or RedisClusterCache based on the cache_config. startup_nodesZREDIS_CLUSTER_NODESNr)getrurrjsonloadsrr)rrZ_env_cluster_nodesrrrrs   zRouter._create_redis_cachercCs|jjdur ||j_dSdS)z Update the redis cache for the router, if none set. Allows proxy user to just do ```yaml litellm_settings: cache: true ``` and caching to just work. N)rr)rrrrr_update_redis_caches zRouter._update_redis_cachecCstd|dgddtD}|dur3t|to||v}t|t}|s3|s3td|d|d|tjjks>|tjkrit|j d|_ tt j t rSt j |j n|j gt _ tt jt rgt j|j dSdS|tjjkst|tjkrt|j |d |_tt jt rt j|jdSdS|tjjks|tjkrt|j |d |_tt jt rt j|jdSdS|tjjks|tjkrt|j |d |_tt jt rt j|jdSdS|tjjks|tjkrt|j id |_tt jt rt j|jdSdS dS) NzRouting strategy: rcSsg|]}|jqSr)value).0srrr sz0Router.routing_strategy_init..zInvalid routing_strategy: 'z'. Valid options: z. Check 'router_settings.routing_strategy' in your config.yaml or the 'routing_strategy' parameter if using the Router SDK directly.) router_cache)r( routing_args)rr rdrr ValueErrorZ LEAST_BUSYr$r'rrrrrrrradd_litellm_callbackZUSAGE_BASED_ROUTINGr*rZUSAGE_BASED_ROUTING_V2r+lowesttpm_logger_v2Z LATENCY_BASEDr)lowestlatency_loggerZ COST_BASEDr(lowestcost_logger)rrrZvalid_strategy_stringsZis_valid_stringZ is_valid_enumrrrrsv                   zRouter.routing_strategy_initcCst|tj|_|tj|_|tj|_|tj|_|tj|_|tj|_|tj|_|tj |_ dSN) factory_functionrZacreate_assistantsZadelete_assistantZaget_assistantsZacreate_threadZ aget_threadZ a_add_messageZ aget_messagesZ arun_threadrrrrrCsz%Router.initialize_assistants_endpointcCs|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtj dd|_ |jtj d d|_ |jtj d d|_ |jtj d d|_ |jtj d d|_ |jtjd d|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_dS)z+Helper to initialize core router endpoints. moderation call_typeanthropic_messagesagenerate_contentaadapter_generate_content aresponses afile_delete afile_content responsesaget_responsesacancel_responsesacompact_responsesadelete_responsesalist_input_items _arealtime_aresponses_websocketacreate_fine_tuning_jobacancel_fine_tuning_jobalist_fine_tuning_jobsaretrieve_fine_tuning_job alist_files aimage_editallm_passthrough_routeN)r0rZ amoderationr5Zaanthropic_messagesr6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFZ afile_listrHrIr1rrr_initialize_core_endpointsNs z!Router._initialize_core_endpointscCsHddlm}m}m}|j|dd|_|j|dd|_|j|dd|_dS)z"Initialize vector store endpoints.r)asearchcreatesearchavector_store_searchr3vector_store_searchvector_store_createN)litellm.vector_stores.mainrKrLrMr0rNrOrP)rrKrLrMrrr"_initialize_vector_store_endpointss z)Router._initialize_vector_store_endpointsc CsTddlm}ddlm}ddlm}ddlm}ddlm}ddlm}ddlm}dd lm}dd lm } dd lm } dd lm } dd lm } |j |dd|_|j |dd|_|j |dd|_|j | dd|_|j |dd|_|j | dd|_|j |dd|_|j | dd|_|j |dd|_|j | dd|_|j |dd|_|j |dd|_dS)z'Initialize vector store file endpoints.racreateadelete)alist) aretrieve)aretrieve_content)aupdaterLdelete)r)retrieve)retrieve_content)ravector_store_file_creater3vector_store_file_createavector_store_file_listvector_store_file_listavector_store_file_retrievevector_store_file_retrieveavector_store_file_contentvector_store_file_contentavector_store_file_updatevector_store_file_updateavector_store_file_deletevector_store_file_deleteN)Zlitellm.vector_store_files.mainrTrVrWrXrYrZrLr]rr^r_rr0r`rarbrcrdrerfrgrhrirjrk) rZavector_store_file_create_fnZavector_store_file_delete_fnZavector_store_file_list_fnZavector_store_file_retrieve_fnZavector_store_file_content_fnZavector_store_file_update_fnZvector_store_file_create_fnZvector_store_file_delete_fnZvector_store_file_list_fnZvector_store_file_retrieve_fnZvector_store_file_content_fnZvector_store_file_update_fnrrr'_initialize_vector_store_file_endpointss`             z.Router._initialize_vector_store_file_endpointscCs\ddlm}m}m}m}|j|dd|_|j|dd|_|j|dd|_|j|dd|_dS) z"Initialize Google GenAI endpoints.r)r6agenerate_content_streamgenerate_contentgenerate_content_streamr6r3rnrmroN)Zlitellm.google_genair6rmrnror0)rr6rmrnrorrr"_initialize_google_genai_endpointss z)Router._initialize_google_genai_endpointscCsdddlm}m}|j|dd|_|j|dd|_ddlm}m}|j|dd|_|j|dd|_d S) z$Initialize OCR and search endpoints.r)aocrocrrqr3rrrKrMrKrMN)Z litellm.ocrrqrrr0Zlitellm.searchrKrM)rrqrrrKrMrrr _initialize_ocr_search_endpointss z'Router._initialize_ocr_search_endpointsc Csddlm}m}m}m}m}m}m}m}m } m } |j |dd|_|j |dd|_|j |dd|_|j |dd|_|j |dd|_|j | d d|_ |j |d d|_|j |d d|_|j |d d|_|j | d d|_ dS)zInitialize video endpoints.r) avideo_contentavideo_generation avideo_list avideo_remix avideo_status video_contentvideo_generation video_list video_remix video_statusrvr3r{rwr|ryr~rurzrxr}N) Zlitellm.videosrurvrwrxryrzr{r|r}r~r0) rrurvrwrxryrzr{r|r}r~rrr_initialize_video_endpointss20 z"Router._initialize_video_endpointsc Csddlm}m}m}m}m}m}m}m}ddl m } |j |dd|_|j |dd|_|j |dd|_|j |dd|_|j |d d|_|j |d d|_|j |d d|_|j |d d|_| D]\} } t || |j | | dq^d S)zInitialize container endpoints.r)acreate_containeradelete_containeralist_containersaretrieve_containercreate_containerdelete_containerlist_containersretrieve_container)_generated_endpointsrr3rrrrrrrN)Zlitellm.containersrrrrrrrrZ#litellm.containers.endpoint_factoryrr0itemssetattr) rrrrrrrrrZcontainer_file_endpointsnamefuncrrr_initialize_container_endpoints6s:( z&Router._initialize_container_endpointscCsL|jtjdd|_|jtjdd|_|jtjdd|_|jtjdd|_dS)z*Initialize Anthropic Skills API endpoints. acreate_skillr3 alist_skills aget_skill adelete_skillN)r0rrrrrr1rrr_initialize_skills_endpointscs z#Router._initialize_skills_endpointsc Csddlm}ddlm}ddlm}ddlm}ddlm}ddlm}ddlm}dd lm}|j |d d |_ |j |d d |_ |j |d d |_ |j |dd |_ |j |dd |_|j |dd |_|j |dd |_|j |dd |_dS)z-Initialize Google Interactions API endpoints.r)acancelrSrU)aget)cancelr[r\r acreate_interactionr3create_interactionaget_interactionget_interactionadelete_interactiondelete_interactionacancel_interactioncancel_interactionN)Zlitellm.interactionsrrTrVrrrLr]r r0rrrrrrrr) rrrrrrrrrrrr"_initialize_interactions_endpointsrs@         z)Router._initialize_interactions_endpointscCsL|||||||||dS)zvHelper to initialize specialized router endpoints (vector store, OCR, search, video, container, skills, interactions).N) rRrlrprt)_override_vector_store_methods_for_routerrrrrr1rrr!_initialize_specialized_endpointss z(Router._initialize_specialized_endpointscCs||dSr/)rJrr1rrrrs z"Router.initialize_router_endpointsrcCsZ|durdS|D]"}t|tstd|dt|dkr*td|dt|dqdS)z3 Validate the fallbacks parameter. NzItem 'z' is not a dictionary.rxz Dictionary 'z%' must have exactly one key, but has z keys.)rrr*len)rrZ fallback_dictrrrrs  zRouter.validate_fallbacksc s|durdSd|v}d|v}d|v}|s|s|rc|jdur g|_d}|jD] }t|tr0|}nq%|durL|jp9||_|jp?||_|jpE||_|j|_nt|j|j|||d}|j |t j |d|vrddl m|jdurug|_tfdd |jD}|s} |j | t j | |D]F} d} | d vrq| d krt|jd } n| d krt|j|j|jd} n | dkrt|jd} | durq|jdurg|_|j | t j | qdS)Ndeployment_affinityresponses_api_deployment_checksession_affinity)r ttl_secondsenable_user_key_affinityenable_responses_api_affinityenable_session_id_affinityencrypted_content_affinityrEncryptedContentAffinityCheckc3s|]}t|VqdSr/)r)r%cbrrr s  z6Router.add_optional_pre_call_checks..)rrrrZprompt_caching)rr) dual_cacherrZenforce_model_rate_limits)r)rrrErrrrrrrrrr+ZElitellm.router_utils.pre_call_checks.encrypted_content_affinity_checkranyrGr&rrrF) rrrrrZexisting_affinity_callbackrZaffinity_callbackZalready_registeredZ ec_callbackpre_call_check _callbackrrrr s             z#Router.add_optional_pre_call_checks deploymentc Csz/t|}|d}tjrtddd}|||d<|WSd|vr-|dddd|d<|WStyG}z tdt ||d}~ww) z returns a copy of the deployment with the api key masked Only returns 2 characters of the api key and masks the rest with * (10 *). rr)Zvisible_prefixZvisible_suffixapi_keyNz **********z+Error occurred while printing deployment - ) copydeepcopyrZredact_user_api_key_infor$Z mask_dict Exceptionrdebugr)rrZ_deployment_copyrZmaskererrrprint_deployments"   zRouter.print_deploymentrmessagesc Ksnz(td|d||d<||d<|j|d<|j||d|jdi|}|WSty6}z|d}~ww) z Example usage: response = router.completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}] zrouter.completion(model=z,..)rroriginal_functionrkwargsNr)rr _completion_update_kwargs_before_fallbacksfunction_with_fallbacksr)rrrrresponserrrr completion8s zRouter.completionc Ksd}d}z|}||d<|j|||dd|d}|d}|dd}|dur:tj|j||f|dd} | |j||d|dd|d}|j||d} | d d} | durh| durh| | j krhd} n| } | |su|j |d i|||j | d |} tjdi| }td |d t|tr|j|||d}|rtjd|ddt|tr|j|||dWS|WSty}ztd |dt|d|dur||||d}~ww)Nrspecific_deploymentrrrrequest_kwargsr silent_modelT)targetrrdaemonrrrrrcachingclientzlitellm.completion(model=) 200 OKrrrResponse output was blocked.messager llm_providermodel_responserinitial_kwargs) Exception r)rget_available_deploymentpop threadingThread_silent_experiment_completionstart_update_kwargs_with_deployment _get_clientr r has_model_id routing_strategy_pre_call_checksrrrrr rrp"_should_raise_content_policy_errorContentPolicyViolationErrorrn_completion_streaming_iteratorrr(_set_deployment_num_retries_on_exception)rrrr model_namer#input_kwargs_for_streaming_fallbackrrthreadpotential_model_clientdynamic_api_key model_client input_kwargsr _should_raiserrrrrKs            zRouter._completioncKsdddlm}||}d|vri|d<d|dd<|dd|dd|d d|d d|S) ze Prepare kwargs for a silent experiment by ensuring isolation from the primary call. rsafe_deep_copyrTis_silent_experimentZlitellm_call_idNlitellm_logging_objstandard_logging_objectZproxy_server_request)'litellm.litellm_core_utils.core_helpersrr)rrr silent_kwargsrrr_get_silent_experiment_kwargss      z$Router._get_silent_experiment_kwargsrc Ksz9|diddrWdSt|}td||jd i|}|jd |ttt t t f|d|WdSt yY}zt d|dt |WYd}~dSd}~ww) zE Run a silent experiment in the background (thread). rrFN%Starting silent experiment for model rr#Silent experiment failed for model : r) r rrrr rrrr r rrerrorrrrrrrrrrrs(  z$Router._silent_experiment_completionstreamcdSr/rrrrrrrrr acompletionzRouter.acompletioncrr/rrrrrrrcrr/rrrrrrrc s"zt||d<||d<||d<|j|d<|j||d|dp!|j}t}||}|r9|j|||dIdHWS|durNt|trN|j d i|IdH}n |j d i|IdH}t} | |} t |j jtj| d|| t|d |WSty} zt t||t| d | d} ~ ww) Nrrrrrpriority)rrrrservicedurationr4 start_timeend_timeparent_otel_spanlitellm_router_instancerZerror_traceback_stroriginal_exceptionr) _acompletionrr rtime_is_prompt_management_model_prompt_management_factoryrrschedule_acompletionasync_function_with_fallbacksasyncio create_taskrasync_service_success_hookrfROUTERrrrD traceback format_exc) rrrrrZrequest_priorityr rrr  _durationrrrrrsX    fallback_itemcomplete_response_object_usagecCsvddlm}tttt|dd}|dur|gng}|dur-t|dr-|jdur-|||j |d}t |d|dS)zCMerge partial-stream usage with fallback-stream usage on the chunk.r)BaseTokenUsageProcessorusageN) usage_objects) Zlitellm.cost_calculatorrrr rmgetattrhasattrrrZcombine_usage_objectsr)rrrrr Zcombined_usagerrr_combine_fallback_usage0s   zRouter._combine_fallback_usagerrcs@ddlmGfdddt}fdd}||S)z Helper to iterate over a streaming response. Catches errors for fallbacks using the router's fallback system rMidStreamFallbackErrorc4eZdZdeffdd ZddZddZZS)zERouter._acompletion_streaming_iterator..FallbackStreamWrapperasync_generatorc>tj|jjjd||_tdrj|_dSdSN)Zcompletion_streamrcustom_llm_provider logging_obj_hidden_params) superrrr*r+_async_generatorr"r,r)rr' __class__rrrrSs zNRouter._acompletion_streaming_iterator..FallbackStreamWrapper.__init__cS|Sr/rr1rrr __aiter__`zORouter._acompletion_streaming_iterator..FallbackStreamWrapper.__aiter__cs|jIdHSr/)r. __anext__r1rrrr4cszORouter._acompletion_streaming_iterator..FallbackStreamWrapper.__anext__)rrrrrr2r4 __classcell__rrr0rFallbackStreamWrapperRs r8c sd}zXz2z 3dHW}|Vq6Wny}zddlm}|jd}tttt|dd}z~ttd}dj }dj }dj } j d <|j s\|jsad <nd d d d|jddgd <j|dj|d||| |ddIdH}t|dr|2z3dHW} | rt| trt| dr| || Vq6ndVWnty} z td| | d} ~ wwWYd}~nd}~wwWtjddxtdrz IdHWnty}z td|WYd}~nd}~ww|dur2t|dr:z |IdHWn)ty1}z td|WYd}~nd}~wwWddSWddSWddSWddS1sVwYdStjddttdrz IdHWnty}z td|WYd}~nd}~ww|durt|drz |IdHWn'ty}z td|WYd}~nd}~wwWdwWdwWdwWdw1swYw)Nrstream_chunk_builderchunksrrrrrrrsystemYou are a helpful assistant. You are given a message and you need to respond to it. You are also given a generated content. You need to respond to the message in continuation of the generated content. Do not repeat the same content. Your response should be in continuation of this text: rolecontent assistantTr@rAprefixrFr)rdisable_fallbacksrrr model_grouprrr2Fallback also failed: )shieldaclose7stream_with_fallbacks: error closing model_response: %s:stream_with_fallbacks: error closing fallback_response: %s) litellm.mainr:r<rr rmr!rr rrrris_pre_first_chunkgenerated_contentr*async_function_with_fallbacks_common_utilsr"rrkr#rrranyioZ CancelScoperI BaseExceptionr) fallback_responseitemrr:complete_response_objectrrFrrrrfallback_errorr%rrrrrrstream_with_fallbacksfs           R    2      zERouter._acompletion_streaming_iterator..stream_with_fallbacksZlitellm.exceptionsr%rn)rrrrr8rWrrVr_acompletion_streaming_iteratorEs   kz&Router._acompletion_streaming_iteratorcsBddlmGfdddt}|fdd}||S)a Sync equivalent of _acompletion_streaming_iterator. Wraps a sync streaming response so that MidStreamFallbackError (raised by CustomStreamWrapper.__next__) triggers the Router's fallback chain instead of surfacing directly to the caller. rr$cr&)zHRouter._completion_streaming_iterator..SyncFallbackStreamWrappersync_generatorcr(r)) r-rrr*r+_sync_generatorr"r,r)rrZr/rrrs zQRouter._completion_streaming_iterator..SyncFallbackStreamWrapper.__init__cSr1r/rr1rrr__iter__r3zQRouter._completion_streaming_iterator..SyncFallbackStreamWrapper.__iter__cSs t|jSr/)nextr[r1rrr__next__s zQRouter._completion_streaming_iterator..SyncFallbackStreamWrapper.__next__)rrrr rr\r^r5rr6r7rSyncFallbackStreamWrappers r_c 3sd}zz D]}|VqWny}zddlm}|jd}tttt|dd}zwttd}dj }dj }dj } j d <|j sW|js\d <nd d d d|jddgd <j|djdi||| d}t|dr|D]} | rt| trt| dr| || VqndVWnty} z td| | d} ~ wwWYd}~nd}~wwWtdrzWnty} z td| WYd} ~ nd} ~ ww|durt|drz|WdSty} z td| WYd} ~ dSd} ~ wwdSdStdrCzWntyB} z td| WYd} ~ nd} ~ ww|durnt|droz|Wwtym} z td| WYd} ~ wd} ~ wwww)Nrr9r;rrrrrrrr=r>r?rBTrCr)rrrr\rGcloserJrKr)rLr:r<rr rmr!rr rrrrrMrNrrr"rrkr#rrrr`rQr) rRrSrr:rTrrFrrrrrUZ close_errr%rrrZ router_selfrrrWs           D      zDRouter._completion_streaming_iterator..stream_with_fallbacksrX)rrrrr_rWrrarrs  \z%Router._completion_streaming_iteratorc sz8|diddrWdSt|}td||jd i|}|jd |ttt |d|IdHWdSt yY}zt d|dt |WYd}~dSd}~ww) z< Run a silent experiment in the background. rrFNrrrrr) r rrrr rrrr rLrrrrrrr_silent_experiment_acompletionTs*  z%Router._silent_experiment_acompletionc sjd}d}i}z;|}||d<t|}t} |j|||dd|dIdH}|}t} | | } t|jjt j | d| | t|d|j ||d|d} | d d} | durjt|j d | |d ||j ||d |d d| d}|j||d }|j|d 7<i| ||j|d |}|d dtjd i|}|dd}|j||dd}|durt|tjr|4IdH |j|||dIdH|IdH}WdIdHn 1IdHswYn|j|||dIdH|IdH}t|tr|j|||d}|rtjd|dd|j|d 7<td|d|j |||dt|trA|j|||dIdHWS|WStj y}z.|didd}|didd}|j!d|d|7_!|dury|"|||d}~wt#y}z)td|dt$|d|dur|j%|d 7<|dur|"|||d}~ww)!z - Get an available deployment - call it with a semaphore over the call - semaphore specific to it's rpm - in the semaphore, make a check against it's local rpm before running Nrrrasync_get_available_deploymentrrr rr)rrrrxrrmax_parallel_requestsrr client_type)rr+r rrrrzlitellm.acompletion(model=r)rrr rrrz$ Deployment Info: request_timeout: z timeout: rrr)&rrrrcrrrrrrfr_track_deployment_metricsrbr_get_async_openai_model_clientrrrrr rr Semaphore&async_routing_strategy_pre_call_checksrprrrrr rnrYTimeoutrrrrr)rrrrrrZ_timeout_debug_deployment_dictrr r r rrrrr _responser+ rpm_semaphorerrrZ deployment_request_timeout_paramZdeployment_timeout_paramrrrrrs          (          zRouter._acompletionrrmetadata_variable_namecCsV|d|j|d<|dttd}|j|dr|}||i||ddS)zm Adds/updates to kwargs: - num_retries - litellm_trace_id - metadata rZlitellm_trace_idNr)rFr)r rrrruuid4_get_model_from_aliasr)rrrrorrrrrs   z&Router._update_kwargs_before_fallbacks exceptionc Csdt|dr |jdur dS|di}|d}|dur0zt||_WdSttfy/YdSwdS)z Set num_retries from deployment litellm_params on the exception. This allows the retry logic in async_function_with_retries to use per-deployment retry settings instead of the global setting. rNr)r"rr rr* TypeError)rrsrrZdep_num_retriesrrrr2s   z/Router._set_deployment_num_retries_on_exceptioncCsX|j}|dip i}|D]\}}|durq|||q||i|dS)z Adds default litellm params to kwargs, if set. Handles inserting this as either "metadata" or "litellm_metadata" depending on the metadata_variable_name rN)rrrrrr)rrrodefaultsZmetadata_defaultskeyr$rrr*_update_kwargs_with_default_litellm_paramsGs z1Router._update_kwargs_with_default_litellm_params function_namec Cs|di}|d}t||d}t|d}||id}|j||d} |d} | |d<| |d<t|td i||d } |j| d | S) z. Handle clientside credential model_infor)rrrxrF)rFridoriginal_model_idrrryrNr)r rr5r1_generate_model_idrWrZupsert_deployment) rrrrxryrZdynamic_litellm_paramsrorF _model_idr|deployment_pydantic_objrrr_handle_clientside_credential\s0   z$Router._handle_clientside_credentialcCs|dipi}t|tr|}n|jdd}|dpg}|dp#g}|s(|r4t|t|}||d<d|vrG|ddurI|d|d<dSdSdS)z Merge tools from deployment litellm_params with request kwargs. When both have tools, concatenate them (deployment tools first, then request tools). tool_choice: use request value if provided, else deployment's. rTrZtoolsZ tool_choiceN)r rrr r)rrZdep_params_rawZ dep_paramsZ dep_toolsZ req_toolsmergedrrr_merge_tools_from_deployment|s  z#Router._merge_tools_from_deploymentcCsr|j||d|di}|dd}|dd}|d}t|dr:|j|||d}|j}|jj}|jj }t |d } | | i ||||d |did } | ry|| d p`g} t | } | D] } | | vrr| | qg| || d <|did }|rd |}|| d pg} || vr| || || d <||d<|j||dd|d<|j|| ddS)a 3 jobs: - Adds selected deployment, model_info and api_base to kwargs["metadata"] (used for logging) - Adds default litellm params to kwargs, if set. - Merges tools from deployment with request (proxy-configured tools + request tools). rryrrapi_baser)rrrrxrz)rryrdeployment_model_nametagslitellm_credential_namez Credential: rdatar)rroN)rr rr6rryr rrrr1rrrr _get_timeoutrw)rrrrxryZdeployment_litellm_model_nameZdeployment_api_baserrroZdeployment_tagsZ existing_tagsZ merged_tagstagZcredential_nameZcredential_tagrrrrs^              z%Router._update_kwargs_with_deploymentcCsF|j||dd}|dd}|dur|dur||jkrd}|S|}|S)a Helper to get AsyncOpenAI or AsyncAzureOpenAI client that was created for the deployment The same OpenAI client is re-used to optimize latency / performance in production If dynamic api key is provided: Do not re-use the client. Pass model_client=None. The OpenAI/ AzureOpenAI client will be recreated in the handler for the llm provider asyncrfrN)rr r)rrrrrrrrrris   z%Router._get_async_openai_model_clientrcCs,|ddp|ddp|jp|jddS)z=Helper to get stream timeout from kwargs or deployment paramsrN)r rr)rrrrrr_get_stream_timeouts  zRouter._get_stream_timeoutcCsH|ddp!|ddp!|ddp!|ddp!|jp!|jdd}|S)zAHelper to get non-stream timeout from kwargs or deployment paramsrNr)r rrrrrrrrr_get_non_stream_timeouts    zRouter._get_non_stream_timeoutcCs8d}|ddr|j||d}|dur|j||d}|S)z6Helper to get timeout from kwargs or deployment paramsNrFr)r rrrrrrr s zRouter._get_timeoutmodelsc sVdtdttffdd }dtdttdtffdd }t|trItdd |DrIg}|D]}||d||d |q/tj |Id H}|St|trtd d |Drg}t |D]\} } |D]}||d|| | d |qcq]tj |Id H} ddt t |D} | D]}t|t r| |d|dq| d|q| Sd Sd S)a6 Async Batch Completion. Used for 2 scenarios: 1. Batch Process 1 request to N models on litellm.Router. Pass messages as List[Dict[str, str]] to use this 2. Batch Process N requests to M models on litellm.Router. Pass messages as List[List[Dict[str, str]]] to use this Example Request for 1 request to N models: ``` response = await router.abatch_completion( models=["gpt-3.5-turbo", "groq-llama"], messages=[ {"role": "user", "content": "is litellm becoming a better product ?"} ], max_tokens=15, ) ``` Example Request for N requests to M models: ``` response = await router.abatch_completion( models=["gpt-3.5-turbo", "groq-llama"], messages=[ [{"role": "user", "content": "is litellm becoming a better product ?"}], [{"role": "user", "content": "who is this"}], ], ) ``` rrc Jzjd||d|IdHWSty$}z|WYd}~Sd}~wwzs Wrapper around self.async_completion that catches exceptions and returns them as a result rNrrrrrrrr1rr_async_completion_no_exceptions> zARouter.abatch_completion.._async_completion_no_exceptionsidxc sRzjd||d|IdH|fWSty(}z ||fWYd}~Sd}~wwrr)rrrrrr1rr*_async_completion_no_exceptions_return_idxI s zLRouter.abatch_completion.._async_completion_no_exceptions_return_idxcs|]}t|tVqdSr/)rrr%rrrrr\ z+Router.abatch_completion..rNcsrr/)rrrrrrrc r)rrrcSsg|]}gqSrr)r%_rrrr'n sz,Router.abatch_completion..rxrr)rr rLrrrallrrgather enumeraterangertuple) rrrrrr_tasksrrrrr;Zfinal_responsesrr1rabatch_completion sN$  zRouter.abatch_completioncsXdtdttffdd }g}|D]}||d||d|qtj|IdH}|S)a Async Batch Completion - Batch Process multiple Messages to one model_group on litellm.Router Use this for sending multiple requests to 1 model Args: model (List[str]): model group messages (List[List[Dict[str, str]]]): list of messages. Each element in the list is one request **kwargs: additional kwargs Usage: response = await self.abatch_completion_one_model_multiple_requests( model="gpt-3.5-turbo", messages=[ [{"role": "user", "content": "hello"}, {"role": "user", "content": "tell me something funny"}], [{"role": "user", "content": "hello good mornign"}], ] ) rrc rrrrr1rrr rz]Router.abatch_completion_one_model_multiple_requests.._async_completion_no_exceptionsrNr)rr rLrrr)rrrrrrZmessage_requestrrr1r-abatch_completion_one_model_multiple_requestsv s" z4Router.abatch_completion_one_model_multiple_requestscrr/rrrrr"abatch_completion_fastest_response rz)Router.abatch_completion_fastest_responsecrr/rrrrrr rc sdd|dD}dtdttttfdtdtdtttt ff fd d }gd t j ffd d }|D]}t |d|||d|} |q5rut jt jdIdH\} | D]} || IdH} | durrd| jd<| Sq\sMt d)z model - List of comma-separated model names. E.g. model="gpt-4, gpt-3.5-turbo" Returns fastest response from list of model names. OpenAI-compatible endpoint. cSsg|]}|qSr)striprrrrr' z=Router.abatch_completion_fastest_response..,rrrrrc spzjd|||d|IdH}|WStjy$td|ty7}z|WYd}~Sd}~ww)zn Wrapper around self.acompletion that catches exceptions and returns them as a result rrrNz4Received 'task.cancel'. Cancelling call w/ model={}.r)rrCancelledErrorrrr r)rrrrresultrr1rrr szRRouter.abatch_completion_fastest_response.._async_completion_no_exceptionstaskcszNz0|IdH}t|ttfr1tdD]}|q|WWz|WSty0YSwWn ty;YnwWz|WdStyOYdSwz|Wwty`Yww)Nz=Received successful response. Cancelling other LLM API calls.) rrprnrrrrKeyErrorr)rrt) pending_tasksrrcheck_response s<      zARouter.abatch_completion_fastest_response..check_responser) return_whenNTZ!fastest_response_batch_completionzAll tasks failedr)splitrr r boolrrrprnrrTaskrrwaitFIRST_COMPLETEDr,) rrrrrrrrrdoneZcompleted_taskrr)rrrr sH     rcrr/rrrrrrrrrrr rzRouter.schedule_acompletioncrr/rrrrrr rc sZt|}tt}t|||d}|jj|dIdHt|j } t} |jj } d} | | kr^|j ||dIdH\} }|jj |j |j| dIdH} | rNnt| IdHt} | | ks1| rz#|jd|||d|IdH}|jdi|jddd i|WSty}zt|d ||d}~ww|jj|j |jd IdHtjd |d d)Nr request_idrrequestFrr r{rZhealth_deploymentsradditional_headers%x-litellm-request-prioritization-usedTrrr%Request timed out while polling queuerrr)rrrrqrJr add_requestr monotonicrr_async_get_healthy_deploymentspollrrrsleeprr,rrrrremove_requestrrl)rrrrrrr  _request_idrSr  curr_time poll_interval make_request_healthy_deploymentsrrmrrrrr sh      rr.c s\t|}tt}t|||d}|jj|dIdHt|j } t} |jj } d} | | kr^|j ||dIdH\} }|jj |j |j| dIdH} | rNnt| IdHt} | | ks1| rz$||i|IdH}t|jtr|jdi|jdddi|WSty}zt|d ||d}~ww|jj|j |jd IdHtjd |d d )NrrFrrrrTrrrrr)rrrrqrJrrrrrrrrrrrrrr,rrrrrrrrl)rrrrrrr rrSr rrrrrrmrrrr_schedule_factoryX sb      zRouter._schedule_factorycCsd|j|d}|dust|dkrdS|dddd}|dus$d|vr&dS|dd}|tjvS)NrrxFrrr/)get_model_listrr rr)_known_custom_logger_compatible_callbacks)rrr litellm_modelsplit_litellm_modelrrrr s  z"Router._is_prompt_management_modelcs|dd}|durtdidttd|\}}tt|}|j|dddg|ddd}|j||d |d  }|d d}t |t oM| d }|d pZ|d d d} |dpg|d dd} |ddpu|d dd} |s| dust | t st d| dt| | durt | tst d| dt| |j||t|d| | | d\}}} hdfdd|D} i| || }||d <||d<||d<| |d <| |d<| |d<|j|d}|dust|dkr|dtjdi|IdHS|jdi|IdHS)Nrr)rZ rules_objr userpromptr?rrrrrrrzlitellm_agent/ prompt_idprompt_variables prompt_labelz*Prompt ID is not set or not a string. Got=z, type=z2Prompt variables is set but not a dictionary. Got=r)rrnon_default_paramsrrr>rZbitbucket_configrrZdotprompt_configZprompt_versionci|] \}}|vr||qSrrr%kvZprompt_management_paramsrr z5Router._prompt_management_factory..rrrrr)r rrrqrvrLiteLLMLoggingrrrrrr startswithr*rrZget_chat_completion_promptrtrrrrrr)rrrrZlitellm_logging_objectZprompt_management_deploymentrrZis_litellm_agent_modelrrrZoptional_paramsZ filtered_dataZ _model_listrrrr s           z!Router._prompt_management_factoryrc Ksvz,||d<||d<|j|d<|d|j|d<|did|i|jdi|}|WSty:}z|d}~ww)NrrrrrrFr)_image_generationr rrrrrrrrrrrrrrimage_generation s zRouter.image_generationc Ks$d}zftd|d||j|dddg|ddd}|j||d |d }|j||d }|j|d 7<|j|d t j dii|||j |d |}|j |d 7<t d|d|WSty} zt d|dt| d|dur|j|d 7<| d} ~ ww)Nr#Inside _image_generation()- model: ; kwargs: rrr?rrrrrxrrrrzlitellm.image_generation(model=rrrr)rrrrrrrirrrrrrr rrr) rrrrrrrrrrrrrr sV      zRouter._image_generationc sz+||d<||d<|j|d<|d|j|d<|j||d|jdi|IdH}|WStyG}ztt||t |d|d}~ww)Nrrrrrr r) _aimage_generationr rrrrrrrDrrrrrraimage_generationD s* zRouter.aimage_generationc s|}ztd|d|t|}|j|dddg|dd|dIdH}|j||d|d }|d }|j||d}|j|d 7<t j dii|||j |d |} |j ||d d} | durt | tjr| 4IdH |j||dIdH| IdH} WdIdHn 1IdHswYn|j||dIdH| IdH} |j|d 7<td|d| WSty} ztd|dt| d|dur|j|d 7<| d} ~ ww)Nrrrrr?rrrrrrxrrerfrdz litellm.aimage_generation(model=rrrr)rrrrcrrrrirrrrrrrrjrkrr rrr) rrrrrr rrrrrnrrrrrY s       (   zRouter._aimage_generationfilec s~z"||d<||d<|j|d<|j||d|jdi|IdH}|WSty>}ztt||t|d|d}~ww)a Example Usage: ``` from litellm import Router client = Router(model_list = [ { "model_name": "whisper", "litellm_params": { "model": "whisper-1", }, }, ]) audio_file = open("speech.mp3", "rb") transcript = await client.atranscription( model="whisper", file=audio_file ) ``` rrrrNr r) _atranscriptionrrrrrrDrr)rrrrrrrrratranscription s( zRouter.atranscriptionc s|}ztd|d|t|}|j|dddg|dd|dIdH}|j||d|d }|j||d}|j|d 7<t j dii|||j |d |} |j ||d d } | durt | tjr| 4IdH |j||dIdH| IdH} WdIdHn 1IdHswYn|j||dIdH| IdH} |j|d 7<td|d| WSty} ztd|dt| d|dur|j|d 7<| d} ~ ww)Nz!Inside _atranscription()- model: rrrr?rrrrrx)rrrrerfrdzlitellm.atranscription(model=rrrr)rrrrcrrrrirrrrrrrrjrkrr rrr) rrrrrr rrrrrnrrrrr s       (   zRouter._atranscriptioninputvoicec s<z||d<||d<|j|dddg|dd|dIdH}|j||d |d }|d |jD]\}}||vrA|||<q4|d krL|||q4|j||d d} |dd} | durk| durk| | j krkd} n| } t j dii|d| i|IdH} | WSt y} zt t||t| d| d} ~ ww)a Example Usage: ``` from litellm import Router client = Router(model_list = [ { "model_name": "tts", "litellm_params": { "model": "tts-1", }, }, ]) async with client.aspeech( model="tts", voice="alloy", input="the quick brown fox jumped over the lazy dogs", api_base=None, api_key=None, organization=None, project=None, max_retries=1, timeout=600, client=None, optional_params={}, ) as response: response.stream_to_file(speech_file_path) ``` rrrrr?rNrrrrrrrfrrr r)rcrrrrrrrr rraspeechrrrrDrr)rrrrrrrrrrrrrrrrrr sd         zRouter.aspeechc s~z"||d<t|d<|j|d<|j||d|jdi|IdH}|WSty>}ztt||t |d|d}~wwNrrrrr r) r_arerankrrrrrrDrrrrrrrrrrarerankg s( zRouter.arerankc s$d}zetd|d||j||dd|dIdH}|j||d|d}|d}|j||d}|j|d7<tj dii||j |d |IdH}|j |d7<t d |d |WSt y}zt d |d t|d |dur|j|d7<|d}~ww)NzInside _rerank()- model: rr)rrrrrrrx)rrzlitellm.arerank(model=rrrr)rrrcrrrrirrrrrr rrr) rrrrrrrrrrrrr| sV      zRouter._arerankis_retry is_fallbackis_asyncc Ksd|dg}z_||d<||d<|d|j|d<|did|i|j|||ddd }|d } |jD]\} } | |vrG| || <q:| dkrR|| | q:t j d ii| ||j d |WSt ys} z| d} ~ ww) Nrr?rrrrrFrrr)rrr) r rrrrrrrrrtext_completionrr) rrrrrrrrrrrrrrrrr s.    &zRouter.text_completionc s|dddur|j||d|j||f|dIdHSz"||d<||d<|j|d<|j||d|jdi|IdH}|WStyY}zt t ||t |d|d}~ww) Nr)rrrrrrrrrr r) r rratext_completion_atext_completionrrrrrrDrr) rrrrrrrrrrrrr s8   zRouter.atext_completionc sztd|d|t|}|j|d|dg|dd|dIdH}|j||d|d}|d }|j||d}|j|d 7<t j dii|||j |d |} |j ||d d } | durt | tjr| 4IdH |j||dIdH| IdH} WdIdHn 1IdHswYn|j||dIdH| IdH} |j|d 7<td|d| WSty} ztd|dt| d|dur|j|d 7<| d} ~ ww)N#Inside _atext_completion()- model: rrr?rrrrrrxrrerfrdzlitellm.atext_completion(model=rrrr)rrrrcrrrrirrrrrrrrjrkrr rrr) rrrrr rrrrrrnrrrrr        (   zRouter._atext_completion adapter_idc sz/||d<||d<|j|d<|d|j|d<|did|i|jdi|IdH}|WStyK}ztt ||t |d|d}~ww) NrrrrrrFr r) _aadapter_completionr rrrrrrrrDrr) rrrrrrrrrrrraadapter_completion7 s*  zRouter.aadapter_completionc sztd|d|t|}|j|dddg|dd|dIdH}|j||d|d }|d }|j||d}|j|d 7<t j dii|||j |d |} |j ||d d} | durt | tjr| 4IdH |j||dIdH| IdH} WdIdHn 1IdHswYn|j||dIdH| IdH} |j|d 7<td|d| WSty} ztd|dt| d|dur|j|d 7<| d} ~ ww)Nz&Inside _aadapter_completion()- model: rrz default textr?rrrrrrx)rrrrerfrdz"litellm.aadapter_completion(model=rrrr)rrrrcrrrrirrrrrrrrjrkrr rrr) rrrrr rrrrrrnrrrrrT rzRouter._aadapter_completioncs*ddlm}|jd||d|IdHS)z Helper function to make a search API call through the router with load balancing and fallbacks. Reuses the router's retry/fallback infrastructure. rSearchAPIRouter)router_instancerNr)&litellm.router_utils.search_api_routerrZasync_search_with_fallbacks)rrrrrrr_asearch_with_fallbacks s  zRouter._asearch_with_fallbacksoriginal_generic_functioncs,ddlm}|jd|||d|IdHS)z Helper function for search API calls - selects a search tool and calls the original function. Called by async_function_with_fallbacks for each retry attempt. rr)rrr Nr)rrZ"async_search_with_fallbacks_helper)rrr rrrrr_asearch_with_fallbacks_helper s  z%Router._asearch_with_fallbacks_helperguardrail_namecs\||d<||d<|j|d<|j||ddtd|d||jd i|IdH}|S) aw Execute a guardrail with load balancing and fallbacks. Args: guardrail_name: Name of the guardrail to execute original_function: The guardrail's execution function (e.g., async_pre_call_hook) **kwargs: Additional arguments passed to the guardrail Returns: Result from the guardrail execution rr rlitellm_metadatarrroz&Inside aguardrail() - guardrail_name: rNr)_aguardrail_helperrrrr)rr rrrrrr aguardrail s zRouter.aguardrailcsP|}|j|d}td|did||d<|di|IdH}|S)a} Helper for aguardrail - selects a guardrail deployment and executes it. Called by async_function_with_fallbacks for each retry attempt. Args: model: The guardrail_name (named 'model' for fallback system compatibility) original_generic_function: The guardrail's execution function **kwargs: Additional arguments r zSelected guardrail deployment: rZ guardrailselected_guardrailNr)get_available_guardrailrrr )rrr rr rrrrrr szRouter._aguardrail_helperrYcsZddlm}fdd|jD}|stdt|dkr#|dStt|||dS)z Select a guardrail deployment using the router's load balancing strategy. Args: guardrail_name: Name of the guardrail to select Returns: Selected guardrail configuration dict rr,csg|] }|dkr|qSrr)r%grrrr'rz2Router.get_available_guardrail..zNo guardrail found with name: rxllm_router_instancerr)&litellm.router_strategy.simple_shuffler-rr*rrrY)rr r-rrrrr s  zRouter.get_available_guardrailc sz.||d<||d<|j|d<|j||ddtd|d||jd i|IdH}|WStyJ}ztt||t |d |d}~ww) Helper function to make a generic LLM API call through the router, this allows you to use retries/fallbacks with litellm router rr rr rz3Inside ageneric_api_call_with_fallbacks() - model: rNr r) (_ageneric_api_call_with_fallbacks_helperrrrrrrrrDrr)rrrrrrrrr!_ageneric_api_call_with_fallbackss2 z(Router._ageneric_api_call_with_fallbacksrcCsvd|vr9|dr9ddlm}z|||d|dd\}}}}|}Wn ty.|}Ynw|d|||d<|S)a Add the deployment model to the endpoint for LLM passthrough route. e.g for bedrock invoke users can pass endpoint as /model/special-bedrock-model/invoke it should be actually sent as /model/us.anthropic.claude-3-5-sonnet-20240620-v1:0/invoke endpointr)rsr*r)rr*r)rrsr rreplace)rrrrrsZstripped_model_namerZreplacement_model_namerrr;_add_deployment_model_to_endpoint_for_llm_passthrough_route5s   zBRouter._add_deployment_model_to_endpoint_for_llm_passthrough_routec s|dd}d}zt|}z|j|||dd|dddIdH}Wn#tyG}z|rA|dd|i|IdHWYd}~WS|d}~ww|j|||d |d } | d} |j| d 7<|j||| d |dii| d |j i|} |j ||dd} | durt | t j r| 4IdH |j||dIdH| IdH} WdIdHn 1IdHswYn|j||dIdH| IdH} |j| d 7<td| d| WSty}ztd|dt|d|dur|j|d 7<|d}~ww)rpassthrough_on_no_deploymentFrrNr)rrrrrrrrx)rrrrrerfrdz'ageneric_api_call_with_fallbacks(model=rrrr)rrrcr rrrrrrrrrrjrkrrr rr) rrr rrrxr rrrrrrnrrrrTs   &   (    z/Router._ageneric_api_call_with_fallbacks_helperc Ks|j}ztd|d|d||j||dd|ddd}|j||dd |d }|d }|j|d 7<d |vrO|d rO|d  |||d <||d <|j |dz t |d d\}} }}Wn t yqd} Ynw|dii|| |j d|} |j|d 7<t|d|d| WSt y} zt|d|dt| d|dur|j|d 7<| d} ~ ww)a Make a generic LLM API call through the router, this allows you to use retries/fallbacks with litellm router Args: model: The model to use original_function: The handler function to call (e.g., litellm.completion) **kwargs: Additional arguments to pass to the handler function Returns: The response from the handler function z&Inside _generic_api_call() - handler: z , model: rrNrrZgeneric_api_callrrrrxrrrp)r*rz(model=rrrr)rrrrr rrrrrrrsrrrr rr) rrrr handler_namerrrrr*rrrrr _generic_api_call_with_fallbackssb       z'Router._generic_api_call_with_fallbacksc Ks\z||d<||d<|j|d<|j||d|jdi|}|WSty-}z|d}~ww)Nrrrrr) _embeddingrrrrrrrrrrrrr embeddings zRouter.embeddingc KsVd}ztd|d||j|||ddd}|j||d|d}|d}|j||dd }|d d}|durI|durI||jkrId} n|} |j |d 7<|j |d t j dii|||j | d |} |j|d 7<td|d| WSty} ztd|dt| d|dur|j|d 7<| d} ~ ww)NzInside embedding()- model: rr)rrrrrrsyncrfrrxrrrrzlitellm.embedding(model=rrrr)rrrrrrrr rrrrr#rrr rrr) rrrrrrrrrrrrrrrr!sb       zRouter._embeddingc s~z"||d<||d<|j|d<|j||d|jdi|IdH}|WSty>}ztt||t|d|d}~wwr) _aembeddingrrrrrrDrrr"rrr aembedding1s( zRouter.aembeddingc sd}ztd|d|t|}|j|||dd|dIdH}|j||d|d}|d}|j||d}|j|d7<t j dii|||j |d |} |j ||d d } | durt | tjr| 4IdH |j||d IdH| IdH} WdIdHn 1IdHswYn|j||d IdH| IdH} |j|d7<td |d| WSty} ztd |dt| d|dur|j|d7<| d} ~ ww)NzInside _aembedding()- model: rr)rrrrrrrrxr%rerfrdzlitellm.aembedding(model=rrrr)rrrrcrrrrirrr'rrrrrjrkrr rrr) rrrrrr rrrrrnrrrrr&Js      (   zRouter._aembeddingc sz'||d<|j|d<|d|j|d<|j||d|jdi|IdH}|WStyC}ztt||t |d|d}~ww)Nrrrrr r) _acreate_filer rrrrrrrDrrrrrr acreate_files( zRouter.acreate_filec sNzyddlm}td|dtj|dddgddd IdH}d td tffd d }g}t |trH| ||n |D] }| ||qJt j |IdH}t |dkrftd|||d} tt|d} | | jd<| WSty} z td|ddt| d|durj|d7<| d} ~ ww)Nr)add_model_file_id_mappingsrrrfiles-api-fake-textr?rrrrrr rrc sddlm}|}j||dd|d}|d}j||d}j|d7<t|dd \}}}}ttt  d } ttt  d } | rP| sTt d t | d } | ret| |d} | |d <d|vrs|d|did<tjdii||j|d|} j||dd} | durt| tjr| 4IdH j|dIdH| IdH} WdIdHn 1IdHswYnj|dIdH| IdH} j|d7<td|d| S)Nrrr)rrrrrxrppurposerz2file and file_purpose are required for create_file)r-)Z file_contentZnew_model_nameZgcs_bucket_namer r*rrrerfrdlitellm.acreate_file(model=rr)rrrrrirrsrr rOr rMrr3r2rrr)rrrrrjrkrrr )rrZ kwargs_copyrrrZstripped_modelr*rr-rZreplace_model_in_jsonl_boolrrnrr rrrcreate_file_for_deployments      (   z8Router._acreate_file..create_file_for_deploymentNo healthy deployments found.)rr;model_file_id_mappingr/, rrrx)!litellm.router_utils.common_utilsr*rrrasync_get_healthy_deploymentsrrrNrrrrrrrr,rsrr) rrrr*rr1tasksrr;r3Zreturned_responserrr0rr(sR    X  zRouter._acreate_filec sz|durddlm}|j|dd}|di|IdHWSddlm}t|}|j|dddg|d d|d IdH}|d }|d } |j||dd |j ||d} |j | d7<t |d d\} } } } |dii|| |j | d|} |j ||dd}|durt|tjr|4IdH|j||dIdH| IdH} WdIdHn 1IdHswYn|j||dIdH| IdH} |j| d7<td| d| WSty}ztd|dt|d|dur|j|d7<|d}~ww)z Create a vector store for a specific model. Args: model: Model name from router config **kwargs: Vector store creation parameters Returns: VectorStoreCreateResponse NrrSavector_store_creater3rzvector-store-api-fake-textr?rrrrrrrxrpr.rerfrdz#litellm.avector_store_create(model=rrrr)rQrTr0Zlitellm.vector_storesrrcrrrrirrsrrrrrjrkrrr rrsrr)rrrrTZ factory_fnZavector_store_create_sdkr rrrrrr*rrnrrrrr83s        (   zRouter.avector_store_createcCsdS)aE Override factory-generated vector store methods with router-aware implementations. This is called after _initialize_vector_store_endpoints() to ensure our custom methods that handle deployment selection and credential injection are used instead of the generic factory-generated ones. Nrr1rrrrs z0Router._override_vector_store_methods_for_routerc sz-||d<|j|d<|d|j|d<tdd}|j|||d|jdi|IdH}|WStyI}ztt ||t |d|d}~ww) Nrrr_acreate_batchrzrr r) r9r rr1rrrrrrDrrrrrrorrrrr acreate_batchs6 zRouter.acreate_batchc ztd|d|t|}|j|dddg|dd|dIdH}|d}|d }|j||d d |j||d }|j|d 7<t |d d\}} }}t j dii|| |j |d|} |j ||dd} | durt| tjr| 4IdH |j||dIdH| IdH} WdIdHn 1IdHswYn|j||dIdH| IdH} |j|d 7<td|d| WSty} z td|d|dt| d|dur|j|d 7<| d} ~ ww)Nz Inside _acreate_batch()- model: rrr+r?rrrrr9rrrxrpr.rerfrdzlitellm.acreate_batch(model=rzlitellm._acreate_batch(model=r4rrr)rrrrcrrrrirrsrr;rrrrrjrkrr rrsrr rrrr rrrrrr*rrnrrrrr9      (   zRouter._acreate_batchc svzt}|dur j|dddgdd|dIdH}n}|dur,tdgdtffd d |dur[t|tr[t|d kr[t j fd d |DddiIdH}n|durot|t rot t|IdH}ntd|durt|t r|WSt|tr|D] }t|t r|WSqrd tdty}zt tt|d|d}~ww)z Iterate through all models in a model group to check for batch Future Improvement - cache the result. Nrzretrieve-api-fake-textr?rr,Router not yet initialized.rc sz]ddlm}|dd}|d}|d}|dur%td||s0t|d\}}}}|}jtt||dd | dd| ddt j d ii|d|i|IdHWSty~}zddl }| |WYd}~dSd}~ww) Nrrrrr*z2Model not found in litellm_params for deployment: rparetrieve_batchrr)rrr rrrsrrrrrr@r print_excr) rrrrr*rZ new_kwargsrr)rreceieved_exceptionsrrrtry_retrieve_batch*sL      z2Router.aretrieve_batch..try_retrieve_batchrcsg|] }tt|qSr)rrXr%rrCrrr'Xs z*Router.aretrieve_batch..return_exceptionsTr2z7Unable to find batch in any model. Received errors - {}r )rr6rrrrXrrrrrrrrir rrDrr)rrrr filtered_model_listresultsrrr)rrBrrCrr@ s|     )        zRouter.aretrieve_batchc sz-||d<|j|d<|d|j|d<tdd}|j|||d|jd i|IdH}|WStyI}ztt ||t |d|d}~ww) zZ Cancel a batch through the router with proper model-to-provider mapping. rrr_acancel_batchrzrNr r) rIr rr1rrrrrrDrrr:rrr acancel_batchs6 zRouter.acancel_batchc r<)Nz Inside _acancel_batch()- model: rrzbatch-api-fake-textr?rrrrrIrrrxrpr.rerfrdzlitellm.acancel_batch(model=rzlitellm._acancel_batch(model=r4rrr)rrrrcrrrrirrsrrJrrrrrjrkrr rrsrrr=rrrrIr>zRouter._acancel_batchcs|j|d}|durtddtffdd tjfdd|DIdH}d gddd d }|D]3}|durd|d durIt|d rIt|d |d <t|d |d <|d|jt|dd durdd|d<q1|S)zQ Return all the batches across all deployments of a model group. rNr?rcs<ztjdii|dIdHWStyYdSw)Nrr)r alist_batchesrrprrrrCs z0Router.alist_batches..try_retrieve_batchcsg|]}|qSrrrDrErrr'rz(Router.alist_batches..rF)objectrfirst_idlast_idhas_morerMrNrrOT) rrrXrrr"r!extendr)rrrrGrHZ final_resultsrr)rrCrrKs0   zRouter.alist_batchesr*cs|j|dd|d|dr?|j|ddr?|j|d|dIdH}|dd|d<|d}|j||d|||di|IdHS) Nrrrr)rrrrr)rr rrcrrr)rrr*rrrrrr)_pass_through_moderation_endpoint_factory*s$    z0Router._pass_through_moderation_endpoint_factory assistantsr4)TrRr2r5r8r=r>r;r<r?r9r:rArBrCrDrErFrGrHrIr@r6rnrmrorNr8r`rbrdrfrhrjrOrPrarcrergrirkrqrrrKrMr7rvr{rwr|ryr~rurzrxr}rrrrrrrraupload_container_fileZupload_container_filealist_container_filesZlist_container_filesaretrieve_container_fileZretrieve_container_fileadelete_container_fileZdelete_container_filerrrrrrrrrrrrcsdvr  d dttdttffdd }|Sdvr1  d dttdttffdd }|S  d dttdttffd d }|S) z Creates appropriate wrapper functions for different API call types. Returns: - A synchronous function for synchronous call types - An asynchronous function for asynchronous call types )r;rnrorOrPrrrMr{r|r~rzr}rrrrNr*rcsjddi|S)Nrr)r r*rr)rrrr sync_wrappers z-Router.factory_function..sync_wrapper)rarcrergrirkcsd||d|S)Nr*rrrrW)rrrvector_store_file_sync_wrappers z?Router.factory_function..vector_store_file_sync_wrappercsdkrjd||d|IdHSdkr%jddi|IdHSdvr6jddi|IdHSdvrHjd|d|IdHSdvrYjddi|IdHSd vrkjd|d|IdHSd kr}jdd d |IdHSd vrjddi|IdHSdvrjd|d|IdHSdvrjd||d|IdHSdvrňjd|d|IdHSdS)NrR)rr*rr2rrs)r`rbrdrfrhrj)rr*)r5r8rArBrCrDrErFrGrHr6rmrqrrrvrwryrurxrrrrrr) rrrrrSrTrUrVZ aretrieve_container_file_contentrIT)rr)r<r=r>r?r@)rNr8)r9r:)rrrr))_pass_through_assistants_endpoint_factoryrQr  _init_vector_store_api_endpointsr_init_containers_api_endpoints_init_responses_api_endpoints _init_interactions_api_endpointsrWr4rrrr async_wrappers            z.Router.factory_function..async_wrapperNN)r rr)rrr4rXrZrarr`rr0Ds4b   }zRouter.factory_functioncsN|r d|vr ||d<|dr|jdd|i|IdHS|di|IdHS)z Initialize the Vector Store API endpoints on the router. If a model is provided in kwargs, use model-based routing to get the deployment credentials. Otherwise, call the original function directly. r*rrNr)r rrrr*rrrrr\Zs   z'Router._init_vector_store_api_endpointscs*|r d|vr ||d<|di|IdHS)z Initialize the Containers API endpoints on the router. Container operations don't need model-based routing, so we call the original function directly with the custom_llm_provider. r*Nrrrcrrrr]ss z%Router._init_containers_api_endpointscsHddlm}||d}|dur||d<|jdd|i|IdHS)z Initialize the Responses API endpoints on the router. GET, DELETE, CANCEL Responses API Requests encode the model_id in the response_id, this function decodes the response_id and sets the model to the model_id. r)ResponsesAPIRequestUtilsZ response_idNrrr)Zlitellm.responses.utilsrdZget_model_id_from_response_idr r)rrrrdmodel_idrrrr^s  z$Router._init_responses_api_endpointscs:|r d|vr ||d<d|vrd|d<|di|IdHS)z Initialize the Interactions API endpoints on the router. GET, DELETE, CANCEL Interactions API Requests don't need model-based routing, so we call the original function directly with the custom_llm_provider. r*geminiNrrrcrrrr_s  z'Router._init_interactions_api_endpointsrcsR|dur|jdur|jd}||jdntd|d||d|IdHS)z@Internal helper function to pass through the assistants endpointNr*rz'custom_llm_provider' must be set. Either via: `Router(assistants_config={'custom_llm_provider': ..})` or `router.arun_thread(custom_llm_provider=..)`rYr)rrr)rrr*rrrrrr[s   z0Router._pass_through_assistants_endpoint_factoryrrErFc  sJtdt|} d} |d} d} |dus| dur ||| d|} d| vr0|j| d<d| vr8d | d<ztd t|d }|r[| || d t |i| IdH}|WSt |t j r|dur|j ||d }|durr| | || d t |i| IdH}|WSd|||}tjd|d|jd|7_nFt |t jr|dur|j ||d }|dur| | || d t |i| IdH}|WSd|||}tjd|d|jd|7_|durO|durOtd|t|tt|d \} }| dur|dur||d} | dur:td|d|t| dr8| jd|d|7_| | | | d t |i| IdH}|WSWn0ty}z#t|}tdt|tt||dIdHt|} WYd}~nd}~wwt| dr| jd|| 7_t| d kr| jd| 7_| )zD Common utilities for async_function_with_fallbacks TracebackNrrT)Zlitellm_routerrrZfallback_depthrzTrying to fallback b/w models)r)fallback_model_grouporiginal_model_group)rrFzmodel={}. context_window_fallbacks={}. fallbacks={}. Set 'context_window_fallback' - https://docs.litellm.ai/docs/routing#fallbackszGot 'ContextWindowExceededError'. No context_window_fallback set. Defaulting to fallbacks, if available.{})msgz {}zmodel={}. content_policy_fallback={}. fallbacks={}. Set 'content_policy_fallback' - https://docs.litellm.ai/docs/routing#fallbackszGot 'ContentPolicyViolationError'. No content_policy_fallback set. Defaulting to fallbacks, if available.{}zinside model fallbacks: rz7No fallback model group found for original model_group=z . Fallbacks=rzlitellm.router.py::async_function_with_fallbacks() - Error occurred while trying to do fallbacks - {} {} Debug Information: Cooldown Deployments={}r r z<. Received Model Group={} Available Model Group Fallbacks={}z Error doing the fallback: {})rrrrr rr r?rrArrContextWindowExceededError(_get_fallback_model_group_from_fallbacksr rrr@rrr"rrrr<r)rrrErrrrFrrrrhriZfallback_failure_exception_strrZis_non_standard_fallback_formatrZ#context_window_fallback_model_group error_messageZ#content_policy_fallback_model_groupZgeneric_fallback_idxZ new_exceptionr rrrrOs.                z1Router.async_function_with_fallbacks_common_utilsc s|d}|dd}|d|j}|d|j}|d|j}|dd}z@|j|||||d |durF|j|i|d|iIdH} n |j|i|IdH} tt j r_t d | t | d d } | WSt y} z|| |||||||IdHWYd} ~ Sd} ~ ww) z Try calling the function_with_retries If it fails after num_retries, fall back to another model group rrEFrrr mock_timeoutN)rrFrrrzAsync Response: r)rZattempted_fallbacks)r rrrr_handle_mock_testing_fallbacksasync_function_with_retriesr isEnabledForrrrr/rrO) rrrrFrErrrrorrrrrrs`    z$Router.async_function_with_fallbackscCst|}|jdur|jdurtj|dd|d|d|jdur5|jdur5tj|dd|d|d|jdurM|jdurOtj|dd|d|ddSdS) a Helper function to raise a litellm Error for mock testing purposes. Raises: litellm.InternalServerError: when `mock_testing_fallbacks=True` passed in request params litellm.ContextWindowExceededError: when `mock_testing_context_fallbacks=True` passed in request params litellm.ContentPolicyViolationError: when `mock_testing_content_policy_fallbacks=True` passed in request params NTr#This is a mock exception for model=z#, to trigger a fallback. Fallbacks=)rrrzF, to trigger a fallback. Context_Window_Fallbacks=zF, to trigger a fallback. Context_Policy_Fallbacks=) r[Z from_kwargsZmock_testing_fallbacksrZInternalServerErrorZmock_testing_context_fallbacksrlZ%mock_testing_content_policy_fallbacksr)rrrFrrrZmock_testing_paramsrrrrps8       z%Router._handle_mock_testing_fallbackscstd|d}|d|j}t|}|d|j}|d|j}|d|j}|d} |d} |d |d p?i} d | vr`t | d t r`|j | d d } | dur`| d t | itd|d| d| d<| | d<z |j| |d|j|g|Ri|IdH} t| ddd} | WSty}z4d}|}t|dd}|durt |tr|} |j|dpd|dIdH\}}d}|jdus|dur| p| d p|d}t||||jd}|dur|} d}|s|j||||||d| | d<| dkr |j||d}ntd| |j|| | ||d}t|IdHt| D]}z8|d| d<| | d<|j|g|Ri|IdH} t| rP| IdH} t| |d| d} | WWYd}~Sty}z?|j||d}| |d}|d}|dur|j||dIdH\}}ng}|j||| ||d}t|IdHWYd}~q)d}~wwt|t j!vrt"|d| |dur|dn| }t"|d||d}~ww)Nz#Inside async function with retries.rrrrrrrr rrFrZmodel_group_sizez/async function w/ retries: original_function - z, num_retries - rattempted_retriesr)rFr)rrtrrrFrsrFrrT)rrall_deploymentsrregular_fallbacksr)rrz#Retrying request with num_retries: )rremaining_retriesrrrvrx)#rrrrrrrrr rrrrr%_handle_mock_testing_rate_limit_error make_callr0rr!rrr"_get_num_retries_from_retry_policyshould_retry_this_error log_retry_time_to_sleep_before_retryrrrr is_async_callablerrZLITELLM_EXCEPTION_TYPESr)rrrrrr rrrrFr _metadatarrrZcurrent_attemptrZdeployment_num_retriesr_all_deploymentsZ_retry_policy_appliesZ_model_group_for_retry_policyZ_retry_policy_retriesrrx_modelr_timeoutZactual_retries_attemptedrrrrqs                  z"Router.async_function_with_retriescsP|d}||i|}t|st|r|IdH}|j||dIdH}|S)z^ Handler for making a call to the .completion()/.embeddings()/etc. functions. rN)rrF)r r rr isawaitableset_response_headers)rrrrrFrrrrrzs   zRouter.make_callcCs|dd}|j|d}d}|dur&t|dkr&ttt|ddd}|durC|durEtd |t j |d d |d |d dSdS)z Helper function to raise a mock litellm.RateLimitError error for testing purposes. Raises: litellm.RateLimitError error when `mock_testing_rate_limit_error=True` passed in request params mock_testing_rate_limit_errorNrrxrrrTzTlitellm.router.py::_mock_rate_limit_error() - Raising mock RateLimitError for model=rrsz , to trigger a rate limit error.)rrrr) rrrrr rr rr rRateLimitError)rrrFrZavailable_modelsrrrrrys,   z,Router._handle_mock_testing_rate_limit_errorrrrvrwc Csd}|durt|trt|}d}|durt|trt|}t|tjr*|dur*|t|tjr6|dur6|t|dd} | durKt| sK| dvrK|t|tjrS|t|t j ri|dkri|durit|dkri|t|t j rv |dkrv||dkr||dS)au 1. raise an exception for ContextWindowExceededError if context_window_fallbacks is not None 2. raise an exception for ContentPolicyViolationError if content_policy_fallbacks is not None 2. raise an exception for RateLimitError if - there are no fallbacks - there are no healthy deployments in the same model group rN status_code)iirxT) rrrrrlrr!Z _should_retryZ NotFoundErrorrrAuthenticationError) rrrrvrrrwZ_num_healthy_deploymentsZ_num_all_deploymentsrrrrr|s@       zRouter.should_retry_this_errorcOst|jg|Ri|S)z~ Sync wrapper for async_function_with_fallbacks Wrapped to reduce code duplication and prevent bugs. )rr)rrrrrrrszRouter.function_with_fallbackscCs@|durdSd}|D]}t|d|kr||}|Sq |S)a@ Returns the list of fallback models to use for a given model group If no fallback model group is found, returns None Example: fallbacks = [{"gpt-3.5-turbo": ["gpt-4"]}, {"gpt-4o": ["gpt-3.5-turbo"]}] model_group = "gpt-3.5-turbo" returns: ["gpt-4"] Nr)rkeys)rrrFrhrSrrrrmsz/Router._get_fallback_model_group_from_fallbackscCsZ|jdurdS|jD] }t|tr*d|vr*|d}t|tr*t|dkr*|dSq dS)zX Returns the first model from the default_fallbacks list, if it exists. Nrr)rrrrr)rfallbackZ default_listrrr_get_first_default_fallback1s   z"Router._get_first_default_fallbackrxcCs|dur t|dkr n|durt|trt|dkrdSd}t|dr-t|jdr-|jj}t|dr5|j}|durEtj||||j d}|Stj|||j d}|S) z Calculate back-off, then retry It should instantly retry only when: 1. there are healthy deployments in the same model group 2. there are fallbacks for the completion call Nrxrrheaderslitellm_response_headers)rxrresponse_headers min_timeout)rxrr) rrrr"rrrrZ_calculate_retry_afterr)rrrxrrrvrrrrrr~>s4  z"Router._time_to_sleep_before_retryc s6ddlm}z|dd}|durtd|dddur"WdS|dddd}|ddd d}|dd ip?i} | d d} |dusN| durQWdSt| trZt| } |j| d } | durgWdS|j| |d } | dd} | dd}| j j }| j j }| dd}| dd}t || d| dur|dur|dur|dur|dur|durWdSt |}|dd}t}|d}tjjj| ||d}g}||||tjjdtjjj| ||d}|||dtjjd|jj||dIdH|WSty}ztdt|WYd}~dSd}~ww)zG Track remaining tpm/rpm quota for model in model_list r)RedisPipelineIncrementOperationrNzstandard_logging_object is NonerrrrFryr{rerreceived_model_nametpmrpmr  deployment_id total_tokens%H-%Mr{current_minuter)rvZincrement_valuerrx)Zincrement_listr zOlitellm.router.Router::deployment_callback_on_success(): Exception occured - {})Zlitellm.types.cachingrr r*rrrget_deploymentget_router_model_inforrrrIrrvstrftimer_TPMr$r rrrRPMrZasync_increment_cache_pipelinerrr)rrcompletion_responser r rrdeployment_namerFryr{deployment_infoZdeployment_model_inforrZtpm_litellm_paramsZrpm_litellm_paramsZtpm_model_infoZrpm_model_infor rdtrZtpm_keyZpipeline_operationsrpm_keyrrrrrqs             z%Router.deployment_callback_on_successc Csd}|dddur n-|dddd}|ddipi}|dd}|dus.|dur0dSt|tr9t|}|durEt||d}|SdS)z Tracks the number of successes for a deployment in the current minute (using in-memory cache) Returns: - key: str - The key used to increment the cache - None: if no key is found NrrrFryr{r)r rrrrI) rrrr r r{rFryrvrrrrs"  z*Router.sync_deployment_callback_on_successc Cstdzw|dd}t|dd}|di}|di}tjjj|d} |d d} d} | dur9tjj | d } | durD| d krD| } n| durO| d krO| } n|j } t |t ru|d } | durcWd St || dt|||| | d}|WStdWd Sty}z|d}~ww)a` 2 jobs: - Tracks the number of failures for a deployment in the current minute (using in-memory cache) - Puts the deployment in cooldown if it exceeds the allowed fails / minute Returns: - True if the deployment should be put in cooldown - False if the deployment should not be put in cooldown z1Router: Entering 'deployment_callback_on_failure'rsNrrrry)rr)rrr{Frr exception_statusrrZtime_to_cooldownzWRouter: Exiting 'deployment_callback_on_failure' without cooldown. No model_info found.)rrr r!rZlitellm_core_utilsZexception_mapping_utilsZ_get_response_headersutilsZ&_get_retry_after_from_exception_headerrrrrHr>r)rrrr r rsrr _model_infoZexception_headersZdeployment_cooldownZheader_cooldownZ_time_to_cooldownrrrrrrrsX         z%Router.deployment_callback_on_failurerc s|dddd}|dddd}|ddipi}|dd}|dus-|dur/dSt|tr8t|}t|} t} | d} tjj j || |d } |j j | d | t jj d IdHdS) z3 Update RPM usage for a deployment rrrNrFryr{rrrx)rvr$r r)r rrrrrvrr_rr$r rZasync_increment_cacherr) rrrr r rrFryr{r rrrrrrrbs2   z+Router.async_deployment_callback_on_failure)rr cCst|S)a Helper to return what the "metadata" field should be called in the request data - New endpoints return `litellm_metadata` - Old endpoints return `metadata` Context: - LiteLLM used `metadata` as an internal field for storing metadata - OpenAI then started using this field for their metadata - LiteLLM is now moving to using `litellm_metadata` for our metadata )r)rrrrr'_get_metadata_variable_name_from_kwargssz.Router._get_metadata_variable_name_from_kwargsc Cszfd|vrdnd}t|jt|d}|D]2\}}||ddfvr'|||<q||krIt|trIi||<||D]\}}|dkrH||||<q:qt|jdkrW|jd|j ||j||d<|WSt yt}z|d }~ww) z When a retry or fallback happens, log the details of the just failed model call - similar to Sentry breadcrumbing r r)Zexception_typeZexception_stringrrrrN) rrrrrrrrrrr) rrrZ _metadata_varZprevious_modelrrZ metadata_kZ metadata_vrrrr}s6    zRouter.log_retryrr cCsX|}|jj||dd}|durd}|jj||ddd|S|d7}|jj||dd|S)zf Update deployment rpm for that minute Returns: - int: request count T)rvr  local_onlyNrxr)rvr$rr)rvr$r)r get_cacheZ set_cache)rrr rZ request_countrrr _update_usages  zRouter._update_usagecCs6|jdurdS|jD]}t|trd|vrdSq dS)NFrT)rrr)rrrrr_has_default_fallbackss   zRouter._has_default_fallbacksrcCs|jrt|jdkr|jdjdkrdS|d|j}|dur=d}|D]}t|d|kr5||}nq#|dur makes the calls concurrency-safe, when rpm limits are set for a deployment Returns: - None Raises: - Rate Limit Exception - If the deployment is over it's tpm/rpm limits Nrstraceback_exceptionr rrryr{r)rrrrZasync_pre_call_checkrrrasync_failure_handlerrrrrrfailure_handlerrr>rrr)rrr r+rrrrrrkHs^     z-Router.async_routing_strategy_pre_call_checksrc s|}tjD]F}t|trLz|j|||||dIdH}WqtyK} z#|durEt|j| t t dt j |j| t fd| d} ~ wwq|S)rrrrrr Nrr)rrrrZasync_filter_deploymentsrrrrrrrrrrr) rrrrr rr+Zreturned_healthy_deploymentsrrrrr!async_callback_filter_deploymentss@     z(Router.async_callback_filter_deploymentsrcCs|g}|D]D\}}t|tr||nt|tr$|t|n|t|t|tr6||qt|trD|t|q|t|qd|}t | }| S)z Helper function to consistently generate the same id for a deployment - create a string from all the litellm params - hash - use hash as id r) rrrrrr!dumpsjoinhashlibsha256encode hexdigest)rrFrpartsrrZ concat_strZ hash_objectrrrr~s        zRouter._generate_model_idr _model_name_litellm_paramsrc srztdi|}tdi||||d}tjD]}|j|dur+|j|||<q|jj}|dur.rTzIgnoring deployment z% as it is not active for environment supported_environmentsrrrezError creating deployment: 1, ignoring and continuing with other deployments.r)rZrWrg model_fieldsrrr ryr{rregister_modelrr*r$deployment_is_active_for_environmentrrr_add_deploymentto_json _add_model_to_list_and_index_maprrrs) rrrrrrrfieldreZ_shared_model_inforrrrr_create_deploymentsj       zRouter._create_deploymentcCs$|jdrdS|jdrdSdS)z Check if the deployment is an auto-router deployment (semantic router). Returns True if the litellm_params model starts with "auto_router/" but NOT "auto_router/complexity_router" (which uses complexity routing). auto_router/complexity_routerFz auto_router/Trrrrrrr_is_auto_router_deployment-s  z!Router._is_auto_router_deploymentcCsddlm}|jj}|jj}|dur|durtd|jj}|dur&td|jj}|dur2td||j|||||d}|j|j vrLtd|jd ||j |j<dS) z Initialize the auto-router deployment. This will initialize the auto-router and add it to the auto-routers dictionary. r)r{Nzzauto_router_config_path or auto_router_config is required for auto-router deployments. Please set it in the litellm_paramszfauto_router_default_model is required for auto-router deployments. Please set it in the litellm_paramszhauto_router_embedding_model is required for auto-router deployments. Please set it in the litellm_params)rauto_router_config_pathauto_router_config default_modelembedding_modelr zAuto-router deployment 3 already exists. Please use a different model name.) /litellm.router_strategy.auto_router.auto_routerr{rrrr*Zauto_router_default_modelZauto_router_embedding_modelrr)rrr{rrrrZ autor_routerrrrinit_auto_router_deployment:sB   z"Router.init_auto_router_deploymentcCs|jdrdSdS)z Check if the deployment is a complexity-router deployment. Returns True if the litellm_params model starts with "auto_router/complexity_router" rTFrrrrr _is_complexity_router_deploymentis z'Router._is_complexity_router_deploymentcCsddlm}|jj}|jj}|dur$|r$|di}|dp#|d}|dur,td||j|||d}|j|jvrDtd |jd ||j|j<dS) z Initialize the complexity-router deployment. This will initialize the complexity-router and add it to the complexity-routers dictionary. rr}NtiersZMEDIUMSIMPLEzcomplexity_router_default_model is required for complexity-router deployments, or configure tiers in complexity_router_config. Please set it in the litellm_params)rrr complexity_router_configzComplexity-router deployment r) ;litellm.router_strategy.complexity_router.complexity_routerr~rrZcomplexity_router_default_modelr r*rr)rrr~rrrZcomplexity_routerrrr!init_complexity_router_deploymentss.     z(Router.init_complexity_router_deploymentcCs|jdusd|jvs|jddurdStdd}|dur td|tvr.tdtd||jdD]}|tvrFtd td|d |q3||jdvrPdSd S) a Function to check if a llm deployment is active for a given environment. Allows using the same config.yaml across multople environments Requires `LITELLM_ENVIRONMENT` to be set in .env. Valid values for environment: - development - staging - production Raises: - ValueError: If LITELLM_ENVIRONMENT is not set in .env or not one of the valid values - ValueError: If supported_environments is not set in model_info or not one of the valid values NrTZLITELLM_ENVIRONMENT)Z secret_namezPSet 'supported_environments' for model but not 'LITELLM_ENVIRONMENT' set in .envz#LITELLM_ENVIRONMENT must be one of z. but set as: z&supported_environments must be one of z for deployment: F)ryrr*rQ)rrZlitellm_environmentZ_envrrrrs,   z+Router.deployment_is_active_for_environmentc Cs0t|}g|_i|_i|_|||D]k}|d}|d}t|t rB| D]\}}t|t rA| drAt |||<q-|di}d|vrV|||} | |d<|dddurzt|dtrz|dD]} | |d<|j||||dqiq|j||||dqtd|d d |D|_dS) Nrr os.environ/ryr{Z organization)rrrrz Initialized Model List cSsh|]}|dqSrrrrrr rz(Router.set_model_list..)rrrrr"_invalidate_model_group_info_cache_invalidate_access_groups_cacherrrrrrrur~r rrrrget_model_namesr) rrZoriginal_model_listrrrrrrZ_idorgrrrrsP          zRouter.set_model_listcCsVddl}|jj}d}d|vr|dd}|tjvrd}|r'|}d}d}d} n#tj|jj|jddd\}}}} |tjvrJt |sJt d||j |jj|jjdurgt|dddurgt|d|j_|jjdur|t|d ddur|t|d |j_d |jvr|j|j|jdd |jjr|j |jj|jd } |jd } | dur| durd | vr| |jvrt|j| <|j| | |jdd |jdgpg} | D])} | di}dD]}||vr||dr||dd}|j|d||<qq|dur |j|||jjd|j|jdr|j|d|j |jdr)|j!|d|S)NrFrTr*rr*zUnsupported provider - rrrrteam_idteam_public_model_nameZ dataSources parameters)rrvrr)rr*r)rr)"osrrrrrrsr Z provider_listr%existsrrrrr!rrrZ add_patternrryr{rrryrrenviron'_initialize_deployment_for_pass_throughrrrr)rrrrZis_prompt_management_modelrrr*rrZ_team_idZ_team_public_model_nameZ data_sourcesZ data_sourcerZ param_keyenv_namerrrrs               zRouter._add_deploymentc Cs|jjdurkddlm}|jjdurt|jj}ni}|dkrP|dp(|jj}|dp1|jj }|dp:|jj }|dusC|durGt d |j |||d n|d pX|jj } |d pa|jj} |j|| | d  dS)a* Optional: Initialize deployment for pass-through endpoints if `deployment.litellm_params.use_in_pass_through` is True Each provider uses diff .env vars for pass-through endpoints, this helper uses the deployment credentials to set the .env vars for pass-through endpoints Tr)passthrough_endpoint_routerNZ vertex_aivertex_projectvertex_locationvertex_credentialsz]vertex_project, and vertex_location must be set in litellm_params for pass-through endpoints.)Z project_idlocationrrr)r*rr)ruse_in_pass_throughZ>litellm.proxy.pass_through_endpoints.llm_passthrough_endpointsrrr!get_credential_valuesr rrrr*Zadd_vertex_credentialsrrZset_pass_through_credentials) rrr*rrcredential_valuesrrrrrrrrr{sN        z.Router._initialize_deployment_for_pass_throughcCs|jj}|r ||r dS|jdd}|j|d|jj}|durF|jjdd}tjD]}|j |}|dur=|||<q-t j ||id|j ||jjd|j|j|S)z Parameters: - deployment: Deployment - the deployment to be added to the Router Returns: - The added deployment - OR None (if deployment already exists) NTrrrr)ryr{rrrr rgrrrr rrrraddr)rr_deployment_model_id _deploymentrZ_model_info_dictrZ field_valuerrradd_deployments*    zRouter.add_deploymentre removal_idxcCs|jD]\}}||kr|d|j|<q||jvr|j|=t|jD]5\}}||vr2||g}|D]}||krD||dq6||q6t|dkrV||j|<q%|j|=q%dS)a" Helper method to update deployment indices after a deployment has been removed from model_list. Parameters: - model_id: str - the id of the deployment that was removed - removal_idx: int - the index where the deployment was removed from model_list rxrN)rrrrrrr)rrerrrrindicesZupdated_indicesrrr(_update_deployment_indices_after_removals$       z/Router._update_deployment_indices_after_removalcCst|j}|j||||dur||j|<n|diddur1||j|dd<|d}|rL||jvrBg|j|<|j||dSdS)a! Helper method to add a model to the model_list and update both indices. Parameters: - model: dict - the model to add to the list - model_id: Optional[str] - the model ID to use for indexing. If None, will try to get from model["model_info"]["id"] Nryr{r)rrrrrrr r)rrrerrrrrrs     z'Router._add_model_to_list_and_index_mapc CszL|jjpd}|j|d}|durD|j|jkrWdSd}|jj}|j}||vrD||}|durD|j||||j ||d|j |d|WSt ym}z|j rgt d|dWYd}~dS|d}~ww)z Add or update deployment Parameters: - deployment: Deployment - the deployment to be added to the Router Returns: - The added/updated deployment rrNrerrzError upserting deployment: r)ryr{rrrrrrrrrrrrr)rrrZ_deployment_on_routerrrZdeployment_fast_mappingrrrrr$s>     zRouter.upsert_deploymentr{cCsld}||jvr |j|}z|dur)|j|}|||j||d|WSWdSty5YdSw)z Parameters: - id: str - the id of the deployment to be deleted Returns: - The deleted deployment - OR None (if deleted deployment not found) Nr)rrrrrrr)rr{Zdeployment_idxrSrrrdelete_deploymentWs     zRouter.delete_deploymentcCsZ||jvr+|j|}|j|}t|trtdi|St|tr"|Stdt|dS)zl Returns -> Deployment or None Raise Exception -> if model found in invalid format zModel invalid format - {}Nr)rrrrrWrr r)rrerrrrrrss     zRouter.get_deploymentcCs8|j|d}|dur dStdi|jjddjddS)zE Returns -> dict of credentials for a given model id rNTrr)rrUrr )rrerrrrget_deployment_credentialss  z!Router.get_deployment_credentialsmodel_group_namecCsb||jvr/|j|}|r/|j|d}t|trtdi|St|tr&|Stdt|dS)z Returns -> Deployment or None Raise Exception -> if model found in invalid format Optimized with O(1) index lookup instead of O(n) linear scan. rzModel Name invalid - {}Nr)rrrrrWrr r)rrrrrrr"get_deployment_by_model_group_names   z)Router.get_deployment_by_model_group_namecCs,|j|d}|dur|j|d}|dur6|j|pg}|r6|d}t|tr/td i|}nt|tr6|}|dur typed dict with max tokens, input cost, etc. Raises: - ValueError -> If model is not mapped yet NrzDeployment not foundry base_modelrrz.Deployment missing valid litellm_params. Got: z, deployment_id: r{unknownrrZazurez Could not identify azure model 'z'. Set azure 'base_model' for accurate max tokens, cost tracking, etc.- https://docs.litellm.ai/docs/proxy/cost_tracking#spend-tracking-for-azure-openai-modelsrz{}/z{}/{}rpr)rr*r rrZrrrrrsrrrrr rrr get_model_inforrrj)rrrr{rrrZlitellm_params_datarrr*rZpotential_modelsZpotential_modelZmodel_info_nameryZuser_model_inforrrrsx       cCs"||jvr|j|}|j|SdS)a% For a given model id, return the model info Returns - dict: the model in list with 'model_name', 'litellm_params', Optional['model_info'] - None: could not find deployment in list Optimized with O(1) index lookup instead of O(n) linear scan. N)rr)rr{rrrrrbs  zRouter.get_model_infocCs,|j|d}|dur dS|d}|j|dS)zT Return list of all models in the same model group as that model id )r{Nrr)rr)rr{ryrrrrget_model_grouprs  zRouter.get_model_groupc Csddlm}d}d}d}ztj|}Wn tyYnwztj|d}Wn ty/Ynwz&|durU|dd}|durUtj|d}|durU|pLi}|tt||}Wn ty_Ynw|durw|durwtt |tt| |}|S|dur}|}|S)z For a given model id, return the model info 1. Check if model_id is in model info 2. If not, check if litellm model name is in model info 3. If not, return None r)_update_dictionaryNrpr) litellm.utilsrrrr rrrrrjr) rrerrryZcustom_model_infoZlitellm_model_name_model_inforZbase_model_inforrrget_deployment_model_info~sR        z Router.get_deployment_model_infouser_facing_model_group_namec Cs:d}d}d}d}|j|d}|durdS|D]f}d} d|vr(|d|kr(d} nd|vr6|j|dur6d} | s9qtdi|d} | j}|di} |di} d} | dur]|dd} | durg| dd} | durq| dd} d}|dur}|d d}|dur| d d}|dur| d d}z| d d}|dur|j|| jd }nd}Wn tyd}Ynwd \}}zt j | j| j d \}}}}Wnt j j y}ztdt|WYd}~nd}~ww|durt j||d }|durg}|di}|dd}t|ddddd|||dd }|dur&tdi||gd|}n7||jvr2|j||dddurU|ddurU|jdusP|d|jkrU|d|_|dddurx|ddurx|jduss|d|jkrx|d|_|dddur|jdus|d|jkr|d|_|dddur|jdus|d|jkr|d|_|dddur|ddurd|_|dddur|ddurd|_|dddur|ddurd|_|dddur|ddurd|_|dddur|ddurd|_|dddur"|ddur"d|_ |dddur7|ddur7|d|_!|dddurJ| durJ|d} |d ddur]|dur]|d }| durm|durid}|| 7}|dur}|duryd}||7}q|dur|dur||_"|dur||_#|dur||_|S) z For a given model group name, return the combined model info Returns: - ModelGroupInfo if able to construct a model group - None if error constructing model group info NrFrTrryrrr{)rer)rrrz.litellm.router.py::get_model_group_info() - {}moderr) rvZ max_tokensmax_input_tokensmax_output_tokensinput_cost_per_tokenoutput_cost_per_tokenZlitellm_providerrsupported_openai_paramsZsupports_system_messages)rF providersrrrr"supports_parallel_function_callingsupports_visionsupports_function_callingsupports_web_searchsupports_url_contextsupports_reasoningrr)$rrr rZ#configurable_clientside_auth_paramsr rrrrrsr* exceptionsBadRequestErrorrrr rget_supported_openai_params ModelMapInfor\rrrrrrr r!r"r#r$r%rrr)rrFrmodel_group_infoZ total_tpmZ total_rpmr&rrZis_matchrZmodel_litellm_paramsZmodel_info_dictZ_deployment_tpmZ_deployment_rpmreryrrrrrZ db_model_inforrrr_set_model_group_infosJ                                     zRouter._set_model_group_infocCsh||jvr-|j|}t|tr|}nt|tr$|ddurdS|d}ndS|j||dS|j||dS)z For a given model group name, return the combined model info Returns: - ModelGroupInfo if able to construct a model group - None if error constructing model group info or hidden model group hiddenTNr)rFr)rrrrr,)rrFrSZ_router_model_grouprrrget_model_group_infos     zRouter.get_model_group_infocsZt}|d}g}g}|j|d}|durdS|D]5}|did}|dd} |dus5| dur6q|tjjj|| |d |tj jj|| |d q||} |j j | d IdH} | duredS| dt |} | t |d} d}| dur| D]}t |tr|durd }||7}q}d}| dur| D]}t |tr|durd }||7}q||fS) z Returns current tpm/rpm usage for model group Parameters: - model_group: str - the received model name from the user (can be a wildcard route). Returns: - usage: Tuple[tpm, rpm] rrNrbryr{rr)r{rr)rr)rvrrr rr_rr$r rrZasync_batch_get_cacherrr)rrFrrZtpm_keysZrpm_keysrrr{rZcombined_tpm_rpm_keysZcombined_tpm_rpm_valuesZtpm_usage_listZrpm_usage_listZ tpm_usagerZ rpm_usagerrrget_model_group_usagesp     zRouter.get_model_group_usage)maxsizecCs ||S)z Cached version of get_model_group_info, uses @lru_cache wrapper This is a speed optimization, since set_response_headers makes a call to get_model_group_info on every request )r.)rrFrrr_cached_get_model_group_infos z#Router._cached_get_model_group_infocs||}|dur|jdur|j}nd}|dur"|jdur"|j}nd}|dur.|dur.iS||IdH\}}i}|durJ||pBd|d<||d<|durZ||pRd|d<||d<|S)Nrx-ratelimit-remaining-tokenszx-ratelimit-limit-tokensx-ratelimit-remaining-requestszx-ratelimit-limit-requests)r1rrr/)rrFr+Z tpm_limitZ rpm_limitZ current_tpmZ current_rpmZ returned_dictrrrget_remaining_model_group_usages. z&Router.get_remaining_model_group_usagecst|trIt|drIt|jtrI|jdi||jdd<|jd}d|vrId|vrI|durI||IdH}|D] \}}|durH|||<q<|S)a Add the most accurate rate limit headers for a given model response. ## TODO: add model group rate limit headers # - if healthy_deployments > 1, return model group rate limit headers # - else return the model's rate limit headers r,rzx-litellm-model-groupr2r3N)rrr"r,rrr4r)rrrFrZremaining_usageheaderr$rrrr1s0    zRouter.set_response_headerscCsR|jt|D]\}}|d}|r&||jvrg|j|<|j||q dS)z Build model_name -> deployment indices mapping for O(1) lookups. This index allows us to find all deployments for a given model_name in O(1) time instead of O(n) linear scan through the entire model_list. rN)rclearrr r)rrrrrrrr_build_model_name_indexUs    zRouter._build_model_name_indexcCsg|_||t|D]:\}}|di}|d}|durB|dd}|di}|||}d|vr||q>|S)z if 'model_name' is none, returns all. Returns list of model id's. Optimized with O(1) or O(k) index lookup when model_name provided, instead of O(n) linear scan. Nryr{r)rrr rrr)rrr9idsrrrrerrr get_model_idss*        zRouter.get_model_ids candidate_idcCs ||jvS)a O(1) membership check for a deployment ID without allocating large lists. Note: Call sites may pass a variable named `model` when it actually contains a deployment ID. This helper expects the deployment ID string. Uses the existing `model_id_to_deployment_index_map` which is kept in sync by `_build_model_id_to_deployment_index_map` and model-list mutation helpers. )r)rr<rrrrs zRouter.has_model_idcCs|sdS||jvs||r|S|jdd}|sdS|D]2}|di}|d}||kp@|o6|d|p@|o@|d|}|rN|d}|rN|SqdS)a Resolve model_name from model_id. This method attempts to find the correct model_name to use with the router so that litellm_params can be automatically injected from the model config. Strategy: 1. First, check if model_id directly matches a model_name or deployment ID 2. If not, search through router's model_list to find a match by litellm_params.model 3. Return the model_name if found, None otherwise Args: model_id: The model_id extracted from decoded video_id (could be model_name or litellm_params.model value) Returns: model_name if found, None otherwise. If None, the request will fall through to normal flow using environment variables. Nrrrr:r)rrrr endswith)rreZ all_modelsrrZ actual_modelmatchesrrrr resolve_model_name_from_model_ids*    z'Router.resolve_model_name_from_model_idteam_model_namercCsH|j||d}|s dS|D]}|did|kr!|dSq dS)z Map a team model name to a team-specific model name. Returns: - deployment id: str - the deployment id of the team-specific model - None: if no team-specific model name is found rrNryrr)rr )rrArrrrrrmap_team_modelszRouter.map_team_modelcCsL|dur|dd|kr||ddkrdS|dur$|d|kr$dSdS)zU Get the team-specific model name if team_id matches the deployment. NryrrTrFr)rrrrrrrshould_include_deploymentsz Router.should_include_deployment model_aliasc Csg}||jvr7|j|}|D]&}|j|}|j|||dr4|dur/|}||d<||q||q|S|durdt|jD]#\}}|j|||drc|dur^|}||d<||q@||q@|S)z Return all deployments of a model name Used for accurate 'get_model_list'. if team_id specified, only return team-specific models Optimized with O(1) index lookup instead of O(n) linear scan. )rrrNr)rrrDrrr) rrrErreturned_modelsrrrZ alias_modelrrr_get_all_deployments s6       zRouter._get_all_deploymentscCsb|pg}g}|D]$}|d}||r%|j||d}|r$||q ||ddq |S)u  Returns all possible model names for the router, including models defined via model_group_alias. If a team_id is provided, only deployments configured with that team_id (i.e. team‐specific models) will yield their team public name. ry)rrrr)rr _is_team_specific_model_get_team_specific_modelr)rrZ deploymentsrrryrArrrr>s    zRouter.get_model_namescCs6|dpi}|dur dS||dkr|dSdS)a Get the team-specific model name if team_id matches the deployment. Args: deployment: DeploymentTypedDict - The model deployment team_id: Optional[str] - If passed, will return router models set with a `team_id` matching the passed `team_id`. Returns: str: The `team_public_model_name` if team_id matches None: If team_id doesn't match or no team info exists ryNrrr)rrrryrrrrIUs  zRouter._get_team_specific_modelrycCst|o|dS)z Check if model info contains team-specific configuration. Args: model_info: Model information dictionary Returns: bool: True if model has team-specific configuration r)rr )rryrrrrHjs zRouter._is_team_specific_modelcCsg}|dur||jvr |S||j|fg}nt|j}|D]/\}}t|tr+|}nt|trCtdi|}|ddur>q|d}nq||j||dq|S)z Helper function to get model list from model alias. Used by `.get_model_list` to get model list from model alias. Nr-Tr)rrEr) rrrrrrrarPrG)rrrFZ alias_itemsrEZ model_valueZ_router_model_nameZ _model_valuerrrget_model_list_from_model_aliasvs*      z&Router.get_model_list_from_model_aliascCsg}|dur||j||d||j|dt|dkr\|j|p&g}|dur?||jvr?|j||p9g}|||dur\|dur\|D]}tdi|}||d<||qI|dure||j 7}|S)z Includes router model_group_alias'es as well if team_id specified, returns matching team-specific models NrBrrrr) rPrGrJrrr rrXrr)rrrrFr Z#potential_team_only_wildcard_modelsrZdeployment_typed_dictrrrrs,       zRouter.get_model_listcCs|jdS)zInvalidate the cached model group info. Call this whenever self.model_list is modified to ensure the cache is rebuilt. N)r1 cache_clearr1rrrrsz)Router._invalidate_model_group_info_cachecCs d|_dS)zInvalidate the cached access groups. Call this whenever self.model_list is modified to ensure the cache is rebuilt. N)rr1rrrrs z&Router._invalidate_access_groups_cachemodel_access_groupc Cs|duo |duo |du}|r|jdur|jSddlm}|t}|j||d}|r_|D]3}|d} | r^| dgp;gD]!} |durR| |krQ|d}|| |q<|d}|| |q|d||kr>|j|||didt||||qtd|qtd|dS) z- Update the router settings. ) rrrrrrrrrrrr)rrrrrrrrzSetting {} is not allowedzUpdated Router settings: N)rSrrrr rrr )rrZ_allowed_settingsZ _int_settingsZ_existing_router_settingsrRZ _casted_valuerrrupdate_settings: s& zRouter.update_settingscCs|dd}t|}|dkr2d|}|jj|d|d}|dur0tj||d|jj|d|d}|S|d kr]|d durM|d }|jj|d|d}|S|d }|jj|d|d}|S|d durs|d }|jj||d}|S|d}|jj||d}|S)a Returns the appropriate client based on the given deployment, kwargs, and client_type. Parameters: deployment (dict): The deployment dictionary containing the clients. kwargs (dict): The keyword arguments passed to the function. client_type (str): The type of client to return. Returns: The appropriate client based on the given client_type and kwargs. ryr{rez{}_max_parallel_requests_clientTrvrr N)r rrrZ_stream_async_clientZ _async_clientZ_stream_client)rvr Z_client)rr rrr4Z set_max_parallel_requests_clientr )rrrrgrer  cache_keyrrrrrm sL      zRouter._get_clientc! std|t|}tztj|d}Wnty5}ztdt ||WYd}~Sd}~wwd}d} d} t |} t } | d} |d| }|j j|d | d pYi}t|D]J\}}|d i}|d i}zO|d d}|dur|d d}|j||d}|p|dd}t|tr|dddurt|dtr||dkr|d }| d||d|7} Wq^Wnty}ztdt |WYd}~nd}~ww|dd}|j j|d | d pd}t|tr-|jdkr-||d||<t|||}t|tr-|dddur-t|dtr-|d|kr-|d } q^|durV|ddurV|d}|durVttd#i||dsV|q^|durtjdurtj|td#i|d\}}}}tj||d}|dur~q^tjj|d}dg}| D]\}} ||vr||vrtd||qq^t!t!|kr | d urt"|d|d urtj#d| |dd t!dkrfd!d"t|D}t!|dkrtj$|}|S)$a Filter out model in model group, if: - model context window < message length. For azure openai models, requires 'base_model' is set. - https://docs.litellm.ai/docs/proxy/cost_tracking#spend-tracking-for-azure-openai-models - filter models above rpm limits - if region given, filter out models not in that region / unknown region - [TODO] function call and model doesn't support function calling z2Starting Pre-call checks for deployments in model=)rzllitellm.router.py::_pre_call_checks: failed to count tokens. Returning initial list of deployments. Got - {}NFrrz:rpm:TrUrryrrrrz%Model={}, Max Input Tokens={}, Got={}zAn error occurs - {}r{rrrallowed_model_region)rrWrr)Z passed_paramsZresponse_formatz1INVALID MODEL INDEX @ REQUEST KWARG FILTERING, k=rpz~litellm._pre_call_checks: Context Window exceeded for given call. No models have context window large enough for this call. {}rcsg|] \}}|vr|qSrr)r%idZinvalid_model_indicesrrr'Y!s z+Router._pre_call_checks..r)%rrrrrZ token_counterrrr rrrvrrrrr rrrrrrsrmaxrwrZZ drop_paramsrsr)rZget_non_default_paramsrrrcrlZ_get_order_filtered_deployments)!rrrrrZ_returned_deploymentsZ input_tokensrZ_context_window_errorZ_potential_error_strZ_rate_limit_errorr rrrZmodel_group_cacherrrrrryreZcurrent_request_cache_localZcurrent_requestrWr*rrrZspecial_paramsrrrrZr_pre_call_checks s                         zRouter._pre_call_checkscCs6||jvrdS|j|}t|tr|}|S|d}|S)z Get the model from the alias. Returns: - str, the litellm model name - None, if model is not in model group alias Nr)rrr)rrZ_itemrrrrrg!s   zRouter._get_model_from_aliascsfdd|jDS)z6 Get the deployment by litellm model. cs g|] }|ddkr|qS)rrrrrprrr'~!s z;Router._get_deployment_by_litellm_model..)r)rrrrpr _get_deployment_by_litellm_modelz!sz'Router._get_deployment_by_litellm_modelrcCsDd}|dur|dp i}|dpi}|dp|d}|dur*||j|dfS||rM|j|d} | durE| jj} | | jddfStd |d |j|d} | durY| }||j vr|j j |d} | rk|| fS|dur||j vr|j |j |d} | r|| fS|j dur|j } |j d | d <|| d d <|| fS|j|d }t|dkr|j|d}ttjrtd|t|dkr|r|}|rtd|d|d|}|j|d }t|dkr|j|d durd|d|}n d|d|}tj||ddtjr|tjvrtj|}||fS)aF Common checks for 'get_available_deployment' across sync + async call. If 'healthy_deployments' returned is None, this means the user chose a specific deployment Returns - str, the litellm model name - List, if multiple models chosen - Dict, if specific model chosen Nrr Zuser_api_key_team_idTrprrzBLiteLLM Router: Trying to call specific deployment, but Model ID :z does not exist in Model ID maprrrrzinitial list of deployments: zModel 'z7' not found. Attempting to use default fallback model 'z'.zYou passed in model=z+. There is no 'model_name' with this stringz1. There are no healthy deployments for this modelrr)r r]rrrrr r*rrrrZget_deployments_by_patternrrrrGrrrrrrrrrr rr rr(Zmodel_alias_map)rrrrrrZrequest_team_idrr rZdeployment_modelZ_model_from_aliasZpattern_deploymentsZupdated_deploymentrZfallback_modelrrrrr!s                z*Router._common_checks_available_deploymentc sT|j|||||d\}}t||d}ttjr!td|t||d}ttjr5td|t|t r<|St ||dIdH}ttjrStd||j ||d}|j |||durit tt|nd||d IdH}|jr|dur|j|t tt|||d }t||||||d IdH}t|d krt|||d IdH} | |S)z Get the healthy deployments for a model. Returns: - List[Dict], if multiple models chosen *OR* - Dict, if specific model chosen rrrrr)rrz'healthy_deployments after team filter: z-healthy_deployments after web search filter: rkNcooldown deployments: rcooldown_deploymentsrrrrr)rrrrrorr rr )rr7rrrrrrr8rrr;_filter_cooldown_deploymentsrrr rLrr\r r.rrrC) rrrrrrr rrarsrrrr6!s           z$Router.async_get_healthy_deploymentsc s|jdkr$|jdkr$|jdkr$|jdkr$|jdkr$|j|||||dSzt|}|j|||||dIdH}|dur@|j}|j}|j||||||d IdH}t|trV|WS| d rft |d krf|d WSt } |jdkr|j dur|j j ||||d IdH} nV|jdkr|jdur|jj ||||d IdH} n>|jdkr|jdur|jj |||||dIdH} n%|jdkrt|||dWS|jdkr|jdur|jj ||dIdH} nd} | durt|||dIdH} | td|d|| d|t } | | } t|jjtj| d|| | d| WStyK}z+t}|durE| dd}|durEtj |j!||fd"t|#|||d}~ww)z Async implementation of 'get_available_deployments'. Allows all cache calls to be made async => 10x perf impact (8rps -> 100 rps). rrrrrr^rrrrrNrrrrrr Z"_encrypted_content_affinity_pinnedrxrrFrrrrFrrrrrrFrrc$get_available_deployment for model: , Selected deployment: for model: 2.async_get_available_deploymentsrrr4r r r rr)$rrrasync_pre_routing_hookrrr6rrr rrr,async_get_available_deploymentsr.r-r-rrCrr rrrrrrfrrrrrrrrr)rrrrrrr pre_routing_hook_responserr rrsr rrrr+rrrrc]"s                          z%Router.async_get_available_deploymentc shzt|}|j|||||dIdH}|dur|j}|j}|j||||||dIdH}t|trQ|di} | dr>|WStj d|didd |d d |j |d } t | d kritj d|d|d d t } |jdkr|jdur|jj|| ||dIdH} n>|jdkr|jdur|jj|| |||dIdH} n%|jdkrt|| |dWS|jdkr|jdur|jj|| dIdH} nd} | durt|||dIdH} | td|d|| t }|| }t|jjtj|d|| |d| WSty3}z+t }|dur-|dd}|dur-t!j"|j#||fd$t|%|||d}~ww)z Async version of get_available_deployment_for_pass_through Only returns deployments configured with use_in_pass_through=True reNrfrr Deployment ryr{C does not support pass-through endpoint (use_in_pass_through=False)rrrrModel z has no deployments configured with use_in_pass_through=True. Please add use_in_pass_through: true to the deployment configurationrrgrrhrrrrircz7async_get_available_deployment_for_pass_through model: , selected deployment: rmrnrr)&rrorrr6rrr rr( _filter_pass_through_deploymentsrr perf_counterrr,rpr-r-rrCrr rrrrrrfrrrrrrrrr)rrrrrrr rqrrpass_through_deploymentsr rrsr rrrr+rrr/async_get_available_deployment_for_pass_through#s                     z6Router.async_get_available_deployment_for_pass_throughcsZ||jvr|j|j|||||dIdHS||jvr+|j|j|||||dIdHSdS)z This hook is called before the routing decision is made. Used for the litellm auto-router to modify the request before the routing decision is made. reN)rror)rrrrrrrrrro#s$    zRouter.async_pre_routing_hookc Cs|j||||d\}}t|tr|St|}t||d}|j||d}|jr3|dur3|j||||d}t|dkrV|j |d} |j j | |d} t||d} t || |j| d |j d kri|jduri|jj||d } nL|j d krut|||d S|j dkr|jdur|jj|||d} n,|j dkr|jdur|jj||||d} n|j dkr|jdur|jj||||d} nd} | durtd|d|j |d} |j j | |d} t||d} t || |j| d td|d|| d|| S)zB Returns the deployment based on routing strategy rrrrrkr`Nrbrr model_idsr rrrZ cooldown_listrrirrrrFrrrrgrrjz, No deployment availablerkrl)rrrrr=rdrr\rr;rget_min_cooldownrbrrget_available_deploymentsr-r-rr,rr r) rrrrrrrr rar}_cooldown_time_cooldown_listrrrrr#s            zRouter.get_available_deploymentcCsL|j||||d\}}t|tr0|di}|dr|Stjd|didd|dd |j|d }t|d krHtjd |d |dd t|} t || d} |j || d}|j ri|duri|j ||||d}t|d kr|j |d} |jj| | d} t || d} t|| |j | d|jdkr|jdur|jj||d}nL|jdkrt|||dS|jdkr|jdur|jj|||d}n,|jdkr|jdur|jj||||d}n|jdkr|jdur|jj||||d}nd}|durtd|d|j |d} |jj| | d} t || d} t|| |j | dtd|d |||S)!a Returns deployments available for pass-through endpoints (based on load balancing strategy) Similar to get_available_deployment, but only returns deployments with use_in_pass_through=True Args: model: Model name messages: Optional list of messages input: Optional input data specific_deployment: Whether to find a specific deployment request_kwargs: Optional request parameters Returns: Dict: Selected deployment configuration Raises: BadRequestError: If no deployment is configured with use_in_pass_through=True RouterRateLimitError: If no pass-through deployments are available r{rrrrryr{rsrrrtrruz has no deployment configured with use_in_pass_through=True. Please add use_in_pass_through: true in the deployment configurationrkr`Nrbrr|r~rrirrrrrrgrz1get_available_deployment_for_pass_through model: z, no available deploymentsrv)rrrr rr(rwrrr=rdrr\r;rrrbrrrr-r-rr,rr r)rrrrrrrrryr rar}rrrrrr)get_available_deployment_for_pass_through8$s                 z0Router.get_available_deployment_for_pass_throughracs6ttjrtd|t|fdd|DS)a Filters out the deployments currently cooling down from the list of healthy deployments Args: healthy_deployments: List of healthy deployments cooldown_deployments: List of model_ids cooling down. cooldown_deployments is a list of model_id's cooling down, cooldown_deployments = ["16700539-b3cd-42f4-b426-6a12a1bb706a", "16700539-b3cd-42f4-b426-7899"] Returns: List of healthy deployments r_cs g|] }|ddvr|qS)ryr{rr%rZ cooldown_setrrr'$s z7Router._filter_cooldown_deployments..)rrrrrrr)rrrarrrrd$s  z#Router._filter_cooldown_deploymentscCs>tdt|ddd|D}tdt|d|S)a Filter out deployments configured with use_in_pass_through=True Args: healthy_deployments: List of healthy deployments Returns: List[Dict]: Only includes a list of deployments that support pass-through z%Filter pass-through deployments from z healthy deploymentscSs$g|]}|diddr|qS)rrFrrrrrr'%s z;Router._filter_pass_through_deployments..zFound z& deployments with pass-through enabled)rrr)rrryrrrrw$s z'Router._filter_pass_through_deploymentsc Cs~z!|didd}|dur|dur|||WdSWdSWdSty>}ztdt|WYd}~dSd}~ww)z7 Tracks successful requests rpm usage. ryr{Nz$Error in _track_deployment_metrics: )r rrrrr)rrr rrerrrrrh%s "z Router._track_deployment_metricscCst|||j|jdS)Nru)r{rr)rrsrFrrrrB%s z(Router.get_num_retries_from_retry_policycCs|j}|dur dSt|tjr|jdur|jSt|tjr%|jdur%|jSt|tjr3|jdur3|jSt|tj rA|j durA|j St|tj rO|j durQ|j SdSdS)a BadRequestErrorRetries: Optional[int] = None AuthenticationErrorRetries: Optional[int] = None TimeoutErrorRetries: Optional[int] = None RateLimitErrorRetries: Optional[int] = None ContentPolicyViolationErrorRetries: Optional[int] = None N) rrrrZAuthenticationErrorAllowedFailsrlZTimeoutErrorAllowedFailsrZRateLimitErrorAllowedFailsrZ'ContentPolicyViolationErrorAllowedFailsr(ZBadRequestErrorAllowedFails)rrsrrrrget_allowed_fails_from_policy)%s2           z$Router.get_allowed_fails_from_policycCsbddlm}|jdur dS|j}||jdg|jd}||_tj|tj |j t ddS)Nr) SlackAlertingZslack)alerting_thresholdZalertingZdefault_webhook_urlz2 Initialized Alerting for litellm.Router ) Z1litellm.integrations.SlackAlerting.slack_alertingrrrZ webhook_urlZslack_alerting_loggerrrr+rZ!response_taking_too_long_callbackrr )rrZrouter_alerting_configZ_slack_alerting_loggerrrrr Q%s"   zRouter._initialize_alertingCustomRoutingStrategycCs t|d|jt|d|jdS)a Sets get_available_deployment and async_get_available_deployment on an instanced of litellm.Router Use this to set your custom routing strategy Args: CustomRoutingStrategy: litellm.router.CustomRoutingStrategyBase rrcN)rrrc)rrrrrset_custom_routing_strategyi%s z"Router.set_custom_routing_strategycCsdt_|jdSr/)rr flush_cacher1rrrr%szRouter.flush_cachecCs*gt_gt_gt_gt_d|_|dSr/)rrrrrrrr1rrrreset%s  z Router.reset)F)rr/)FFF)T)rRrb)NNNN)NNNNN)NF)rN)NNN)NNFN)NNF(rrrrr__annotations__rr rrrtenacityrr'rr*rr rrrrr`rXr rrTrerYrrfloatr rar^rSr]rhrRr staticmethodrrrrrrr#rdrrrJrRrlrprtrrrrrrrr rrprnrrrrrrLrrkrmr#rYrrbrrrrrwrWrrrrirrrrrrrrrrrrrrrrMrrrrrrrrrrr r rrrrrrr ror#r!r'r&rNr)r(r8rrir;r9r@rJrIrKrQr0r\r]r^r_rr[rOr"wraprrprqrzryr|rrmrr~rrrrrr}rzrrrrrrrrkrr~rrZrrrrrrrrrrrrrrrrrr r*rrrrjrr\r,r.r/rrr1r4rr7r8r;rr@rCrDrGrrIrHrJrrrrOrPrSrTrr\rrr]rr6rcrzr|rorrrdrwrhrBrr rVrrrrrrrrsV       !$%&'*+,-01234 7:=@CDLMNOPSTU F  "     O HH (-$  j    e        5      0      F   " & \  /    S    B   A   d .H*FV/ ) %J F     #     Q G  <  G  ` Rx"P6r @6/0& H    3#L"% &C 3V / -(4<+'3 M  ` 8i I !$$ "6$5 ' * .  !3;E ~g(*$ ( r)rrenumrrr!rrrrrMr functoolsrtypingrrrrr r r r r rrrrPZhttpxrrZpydanticrZtyping_extensionsrrZlitellm.litellm_core_utilsZ2litellm.litellm_core_utils.exception_mapping_utilsrZlitellm._loggingrZ litellm._uuidrZlitellm.caching.cachingrrrrZlitellm.constantsrZ"litellm.integrations.custom_loggerrZ#litellm.litellm_core_utils.asyncifyrrrrZ,litellm.litellm_core_utils.coroutine_checkerr Z.litellm.litellm_core_utils.credential_accessorr!Z%litellm.litellm_core_utils.dd_tracingr"Z*litellm.litellm_core_utils.litellm_loggingr#rZ0litellm.litellm_core_utils.sensitive_data_maskerr$Z$litellm.llms.openai_like.json_loaderr%Z&litellm.router_strategy.budget_limiterr&Z"litellm.router_strategy.least_busyr'Z#litellm.router_strategy.lowest_costr(Z&litellm.router_strategy.lowest_latencyr)Z&litellm.router_strategy.lowest_tpm_rpmr*Z)litellm.router_strategy.lowest_tpm_rpm_v2r+rr-Z)litellm.router_strategy.tag_based_routingr.Z/litellm.router_utils.add_retry_fallback_headersr/r0Z litellm.router_utils.batch_utilsr1r2r3Z/litellm.router_utils.client_initalization_utilsr4Z2litellm.router_utils.clientside_credential_handlerr5r6r5r7r8Z#litellm.router_utils.cooldown_cacher9Z&litellm.router_utils.cooldown_handlersr:r;r<r=r>Z,litellm.router_utils.fallback_event_handlersr?r@rAZ*litellm.router_utils.get_retry_from_policyrBr{Z!litellm.router_utils.handle_errorrCrDZ>litellm.router_utils.pre_call_checks.deployment_affinity_checkrEZ;litellm.router_utils.pre_call_checks.model_rate_limit_checkrFZDlitellm.router_utils.pre_call_checks.prompt_caching_deployment_checkrGZ>litellm.router_utils.router_callbacks.track_deployment_metricsrHrIZlitellm.schedulerrJrKZlitellm.types.llms.openairLrMrNrOZlitellm.types.routerrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdreZlitellm.types.servicesrfZlitellm.types.utilsrgrhrirjr*rkrlrmrrnrorprqrrrsrtrurvrwZ&router_utils.pattern_match_deploymentsryZopentelemetry.tracerz_Spanrr{r|rr~Enumrrrrrrs   8                             `   0