o ưi@sdZddlZddlmZmZmZddlmZmZmZm Z m Z m Z ddl Z ddl mZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z m!Z!ddl"m#Z#ddl"m#Z$ddl"m%Z%m&Z&dZ'GdddZ(GdddeZ)dS)a Provider budget limiting Use this if you want to set $ budget limits for each provider. Note: This is a filter, like tag-routing. Meaning it will accept healthy deployments and then filter out deployments that have exceeded their budget limit. This means you can use this with weighted-pick, lowest-latency, simple-shuffle, routing etc Example: ``` openai: budget_limit: 0.000000000001 time_period: 1d anthropic: budget_limit: 100 time_period: 7d ``` N)datetime timedeltatimezone)AnyDictListOptionalTupleUnion)verbose_router_logger) DualCache)RedisPipelineIncrementOperation) CustomLoggerSpan)duration_in_seconds)_get_tags_from_request_kwargs)%_get_prometheus_logger_from_callbacks)AllMessageValues)DeploymentTypedDictLiteLLM_Params RouterErrors) BudgetConfig)GenericBudgetConfigTypeStandardLoggingPayloadc@seZdZdZdZdeeeffddZdedefdd Z dedefd d Z dede fd d Z d dededefddZ ddZddZddZddZdefddZdeeeffddZdeeeffddZdS)!_LiteLLMParamsDictViewz Lightweight attribute view over `litellm_params` dict. This avoids pydantic construction in request hot-path while preserving attribute-style access used by `litellm.get_llm_provider(...)`. _paramsparamscCs ||_dSNr)selfrr!]/home/app/Keep/.python/lib/python3.10/site-packages/litellm/router_strategy/budget_limiter.py__init__6 z_LiteLLMParamsDictView.__init__keyreturncC |j|Srrgetr r%r!r!r" __getattr__9 z"_LiteLLMParamsDictView.__getattr__cCr'rr(r*r!r!r" __getitem__<r,z"_LiteLLMParamsDictView.__getitem__cCs ||jvSrrr*r!r!r" __contains__?r$z#_LiteLLMParamsDictView.__contains__NdefaultcCs|j||Srr()r r%r/r!r!r"r)Bsz_LiteLLMParamsDictView.getcC |jSr)rkeysr r!r!r"r1Er$z_LiteLLMParamsDictView.keyscCr0r)rvaluesr2r!r!r"r3Hr$z_LiteLLMParamsDictView.valuescCr0r)ritemsr2r!r!r"r4Kr$z_LiteLLMParamsDictView.itemscC t|jSr)iterrr2r!r!r"__iter__Nr$z_LiteLLMParamsDictView.__iter__cCr5r)lenrr2r!r!r"__len__Qr$z_LiteLLMParamsDictView.__len__cCr5rdictrr2r!r!r"r;Tr$z_LiteLLMParamsDictView.dictcCr5rr:r2r!r!r" model_dumpWr$z!_LiteLLMParamsDictView.model_dumpr)__name__ __module__ __qualname____doc__ __slots__rstrrr#r+r-boolr.r)r1r3r4r7intr9r;r<r!r!r!r"r,src@seZdZ dNdedeedeeeeee e e fffddZ  dOde ded eee d eed eed eef d dZdee e e fdee e e fde e efde e efdeee de e efdee d eee e e fe ffddZ dNdee e e fd ee d eee e e efe e efeee ffddZde deded efddZde de dededed ef d d!Zde ded"efd#d$Zd%d&Zd'ede de defd(d)Zd*d+Zd,d-Zd.d/Zd0e d eefd1d2Zd3e d eefd4d5Zd6e d eefd7d8Z d9e d ee fd:d;Z!d3e dd?Z"d3e d eefd@dAZ#d3e d ee fdBdCZ$d3e d'efdDdEZ%e& dNdeedeeeeee e e fffdFdGZ'dHdIZ( dNdeeeeee e e fffdJdKZ)dLdMZ*dS)PRouterBudgetLimitingN dual_cacheprovider_budget_config model_listcCsh||_g|_t|||_d|_d|_||j |d| t t j tr2t j|dSdS)N)rH)rFredis_increment_operation_queueasyncio create_task(periodic_sync_in_memory_spend_with_redisrGdeployment_budget_configtag_budget_config_init_provider_budgets_init_deployment_budgets_init_tag_budgets isinstancelitellm callbackslistZlogging_callback_managerZadd_litellm_callback)r rFrGrHr!r!r"r#\s  zRouterBudgetLimiting.__init__modelhealthy_deploymentsmessagesrequest_kwargsparent_otel_spanr&c st|tr |g}t|dkr|Sg}|j||dIdH\}}} } t|dkrt|jj||dIdH} | p:dgt|} i} t|D]\}}t| |pKd| |<qA|j||| | | |t |dd\}}t|dkrrt t j j d||S|S) z Filter out deployments that have exceeded their provider budget limit. Example: if deployment = openai/gpt-3.5-turbo and openai spend > openai budget limit then skip this deployment r)rWrYN)r1rZrY)rWprovider_configsdeployment_configsdeployment_providers spend_mappotential_deployments request_tags: )rRr;r80_async_get_cache_keys_for_router_budget_limitingrFasync_batch_get_cache enumeratefloat$_filter_out_deployments_above_budgetr ValueErrorrZ+no_deployments_with_provider_budget_routingvalue)r rVrWrXrYrZra cache_keysr]r^r_Z_current_spendsZcurrent_spendsr`idxr%deployment_above_budget_infor!r!r"async_filter_deploymentstsX      z-RouterBudgetLimiting.async_filter_deploymentsrar]r^r_r`rbc Cs$d}t|D]\} } d} |jrb| t|kr|| } n|| } | |vrb|| } | jdur.q|d| d| jd}|j| || jd| jrb|| jkrbd| d |d | j}||d 7}d } q|jr| r| d }| dpri}|d}| did}||vr||} |d|d| jd}| jr|| jkrd|d|d|d |d | j }t |||d 7}d } q|j r| r|D]<}| |}|r|d|d|jd}|jr||jkrd|d|d|j}t |||d 7}d } qq| r | | q||fS)a Filter out deployments that have exceeded their budget limit. Follow budget checks are run here: - Provider budget - Deployment budget - Request tags budget Returns: Tuple[List[Dict[str, Any]], str]: - A tuple containing the filtered deployments - A string containing debug information about deployments that exceeded their budget limit. TNprovider_spend::r[providerspend budget_limitzExceeded budget for provider rcz >=  FZ model_namelitellm_paramsrV model_infoiddeployment_spend:z+Exceeded budget for deployment model_name: z, litellm_params.model: , model_id: tag_spend:zExceeded budget for tag='z ', tag_spend=z, tag_budget_limit=)rfrGr8 _get_llm_provider_for_deployment max_budgetr)budget_duration+_track_provider_remaining_budget_prometheusrMr debugrN_get_budget_config_for_tagappend)r rarWr]r^r_r`rbrmrl deploymentZis_within_budgetrsconfig current_spendZ debug_msgZ _model_name_litellm_paramsZ_litellm_model_namemodel_id_tag_tag_budget_configZ _tag_spendr!r!r"rhsz       $     z9RouterBudgetLimiting._filter_out_deployments_above_budgetcsg}i}i}g}|D]z}|jr<||}|||dur<||} | dur<| jdur<| ||<|d|d| j|jre|did} | dure|| } | dure| || <|d| d| j|jrt |d} | D]} | | } | r|d| d| jqoq ||||fS) aB Returns list of cache keys to fetch from router cache for budget limiting and provider and deployment configs Returns: Tuple[List[str], Dict[str, GenericBudgetInfo], Dict[str, GenericBudgetInfo], List[Optional[str]]]: - List of cache keys to fetch from router cache for budget limiting - Dict of provider budget configs `provider_configs` - Dict of deployment budget configs `deployment_configs` - List of resolved providers aligned by deployment index `deployment_providers` Nrprqrxryrzr\r|) rGr}r_get_budget_config_for_providerrrMr)!_get_budget_config_for_deploymentrNrr)r rWrYrkr]r^r_rrs budget_configrrbrrr!r!r"rdsT      zERouterBudgetLimiting._async_get_cache_keys_for_router_budget_limitingstart_time_key current_time ttl_secondscs@|j|IdH}|dur|jj|||dIdH|St|S)z Checks if the key = `provider_budget_start_time:{provider}` exists in cache. If it does, return the value. If it does not, set the key to `current_time` and return the value. Nr%rjttl)rFasync_get_cacheasync_set_cacherg)r rrr budget_startr!r!r"_get_or_set_budget_start_time_s  z2RouterBudgetLimiting._get_or_set_budget_start_time spend_key response_costcs6|jj|||dIdH|jj|||dIdH|S)a Handle start of new budget window by resetting spend and start time Enters this when: - The budget does not exist in cache, so we need to set it - The budget window has expired, so we need to reset everything Does 2 things: - stores key: `provider_spend:{provider}:1d`, value: response_cost - stores key: `provider_budget_start_time:{provider}`, value: current_time. This stores the start time of the new budget window rN)rFr)r rrrrrr!r!r"_handle_new_budget_windowps  z.RouterBudgetLimiting._handle_new_budget_windowrcs:|jjj|||dIdHt|||d}|j|dS)a Increment spend within existing budget window Runs once the budget start time exists in Redis Cache (on the 2nd and subsequent requests to the same provider) - Increments the spend in memory cache (so spend instantly updated in memory) - Queues the increment operation to Redis Pipeline (using batched pipeline to optimize performance. Using Redis for multi instance environment of LiteLLM) rN)r%Zincrement_valuer)rFin_memory_cacheZasync_incrementr rIr)r rrrZ increment_opr!r!r""_increment_spend_in_current_windows  z7RouterBudgetLimiting._increment_spend_in_current_windowcsTtd|dd}|durtd|dd}t|dd}|d id d}|dur4td ||} | rUd |d | j} d|} |j| | | |dIdH||} | rvd|d | j} d|}|j| | ||dIdHt |}t |dkr|D]%}| |}|rd|d |j}d|}|j||||dIdHqdSdS)z)Original method now uses helper functionsz/in RouterBudgetLimiting.async_log_success_eventZstandard_logging_objectNz$standard_logging_payload is requiredrrrrorwcustom_llm_providerzcustom_llm_provider is requiredrprqprovider_budget_start_time:)rrrrrzzdeployment_budget_start_time:r|ztag_budget_start_time:) r rr)rirBrr_increment_spend_for_keyrrr8r)r kwargsZ response_obj start_timeend_timeZstandard_logging_payloadrrrrrrrMZdeployment_spend_keyZdeployment_start_time_keyrbrrZ_tag_spend_keyZ_tag_start_time_keyr!r!r"async_log_success_eventsh            z,RouterBudgetLimiting.async_log_success_eventrc s|jdurdSttj}t|j}|j|||dIdH}|dur2|j|||||dIdH}n.|||krKt d|j|||||dIdH}n|||}t |} |j ||| dIdHt d|d|dS)N)rrr)rrrrrz,Budget window expired - resetting everything)rrrzIncremented spend for z by ) rrnowrutc timestamprrrr rrDr) r rrrrrrrZremaining_timeZttl_for_incrementr!r!r"rsF       z-RouterBudgetLimiting._increment_spend_for_keyc sr z|IdHttIdHWn$ty7}ztdt|ttIdHWYd}~nd}~wwq)z Handler that triggers sync_in_memory_spend_with_redis every DEFAULT_REDIS_SYNC_INTERVAL seconds Required for multi-instance environment usage of provider budgets TNzError in periodic sync task: ) _sync_in_memory_spend_with_redisrJsleepDEFAULT_REDIS_SYNC_INTERVAL Exceptionr errorrBr er!r!r"rLsz=RouterBudgetLimiting.periodic_sync_in_memory_spend_with_redisc sz'|jjs WdStd|jt|jdkr#t|jjj|jdg|_WdSt yE}zt dt |WYd}~dSd}~ww)a How this works: - async_log_success_event collects all provider spend increments in `redis_increment_operation_queue` - This function pushes all increments to Redis in a batched pipeline to optimize performance Only runs if Redis is initialized Nz.Pushing Redis Increment Pipeline for queue: %sr)Zincrement_list*Error syncing in-memory cache with Redis: ) rF redis_cacher rrIr8rJrKZasync_increment_pipelinerrrBrr!r!r"#_push_in_memory_increments_to_redis#s*  z8RouterBudgetLimiting._push_in_memory_increments_to_redisc sz|jjdur WdS|IdHg}|jdur4|jD]\}}|dur'q|d|d|jq|jdurT|jD]\}}|durGq>|d|d|jq>|jdurt|jD]\}}|durgq^|d|d|jq^|jjj |dIdH}t |t r|D]$\}}|dur|jj j |t|dIdHtd|d |qWdSWdSty} ztd t| WYd} ~ dSd} ~ ww) a  Ensures in-memory cache is updated with latest Redis values for all provider spends. Why Do we need this? - Optimization to hit sub 100ms latency. Performance was impacted when redis was used for read/write per request - Use provider budgets in multi-instance environment, we use Redis to sync spend across all instances What this does: 1. Push all provider spend increments to Redis 2. Fetch all current provider spend from Redis to update in-memory cache Nrprqrzr|)Zkey_list)r%rjzUpdated in-memory cache for rcr)rFrrrGr4rrrMrNrerRr;rrrgr rrrrB) r rkrsrrtagZ redis_valuesr%rjrr!r!r"rAs\        z5RouterBudgetLimiting._sync_in_memory_spend_with_redisrcC|jdurdS|j|dSr)rMr))r rr!r!r"rs z6RouterBudgetLimiting._get_budget_config_for_deploymentrscCrr)rGr))r rsr!r!r"rs z4RouterBudgetLimiting._get_budget_config_for_providerrcCrr)rNr))r rr!r!r"rs z/RouterBudgetLimiting._get_budget_config_for_tagrcCsz;|dpi}t|tr|jpd}|}nt|tr&|dp d}t|}nd}ti}tjt||d\}}}}W|St yMt d|YdSw)NrwrorV)rVrwz+Error getting LLM provider for deployment: ) r)rRrrVr;rrSZget_llm_providerrBrr r)r rZdeployment_litellm_paramsrVZprovider_resolution_params_rr!r!r"r}s.    z5RouterBudgetLimiting._get_llm_provider_for_deploymentrtrucCs"t}|r|j|||ddSdS)z Optional helper - emit provider remaining budget metric to Prometheus This is helpful for debugging and monitoring provider budget limits. rrN)rZtrack_provider_remaining_budget)r rsrtruZprometheus_loggerr!r!r"rs  z@RouterBudgetLimiting._track_provider_remaining_budget_prometheuscsn||}|dur dSd|d|j}|jjr$|jj|IdH}n |j|IdH}|dur5t|SdS)ab GET the current spend for a provider from cache used for GET /provider/budgets endpoint in spend_management_endpoints.py Args: provider (str): The provider to get spend for (e.g., "openai", "anthropic") Returns: Optional[float]: The current spend for the provider, or None if not found Nrprqr[)rrrFrrrg)r rsrrrr!r!r"_get_current_provider_spends z0RouterBudgetLimiting._get_current_provider_spendcs||}|dur dSd|d|j}|jjr$|jj|IdH}n |j|IdH}|dur3dSttjt |d S)Nrprq)seconds) rrrFrZ async_get_ttlrrrrr isoformat)r rsrrrr!r!r"%_get_current_provider_budget_reset_ats z:RouterBudgetLimiting._get_current_provider_budget_reset_atcsd|d|j}d|}d}|jdurt|j}|j|IdH}|durr?r rr;r rrrrBrr#rrrnrrgr rhrdrDrrrrrrLrrrrrr}rrrr staticmethodrrOrPrQr!r!r!r"rE[s$   K     ^    E   ; 3C        #rE)*r@rJrrrtypingrrrrr r rSZlitellm._loggingr Zlitellm.caching.cachingr Zlitellm.caching.redis_cacher Z"litellm.integrations.custom_loggerrrZ*litellm.litellm_core_utils.duration_parserrZ)litellm.router_strategy.tag_based_routingrZ'litellm.router_utils.cooldown_callbacksrZlitellm.types.llms.openairZlitellm.types.routerrrrZlitellm.types.utilsrrrrrrrEr!r!r!r"s(          /