o ưi @szddlZddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZmZddlmZddlmZmZddlmZmZmZmZddlTdd lm Z m!Z!dd l"m#Z#erodd l$m%Z%neZ%da&d d Z'GdddeZ( dde e)de*de e)de+fddZ,de+de e)e)ffddZ-de e)de)de.fddZ/de e)de e)e)ffddZ0dS) N)datetime timedelta) TYPE_CHECKINGAny AwaitableCallableDictListLiteralOptionalTupleUnioncast) print_verboseverbose_logger CustomLogger) get_litellm_metadata_from_kwargs&get_metadata_variable_name_from_kwargs) LiteLLM_DeletedVerificationTokenLiteLLM_TeamTableLiteLLM_UserTableUserAPIKeyAuth)*)_sanitize_prometheus_label_name _sanitize_prometheus_label_valueStandardLoggingPayload)AsyncIOSchedulercCstdur ddlm}|atS)z Get cached get_end_user_id_for_cost_tracking function. Lazy imports on first call to avoid loading utils.py at import time (60MB saved). Subsequent calls use cached function for better performance. Nr!get_end_user_id_for_cost_tracking)"_get_end_user_id_for_cost_trackingZ litellm.utilsr rr"V/home/app/Keep/.python/lib/python3.10/site-packages/litellm/integrations/prometheus.py)_get_cached_end_user_id_for_cost_tracking4s r$c@s~eZdZddZdeeeeffddZdedefddZ d ede e fd d Z d ed eede e fd dZdedeeeeffddZd ed eefddZdeddfddZd edeedeeddfddZdededdfddZd efdd Zd!eeeefddfd"d#Zd edefd$d%Zd&d'Zd edeefd(d)Zd*d+Zd,ed-e ed.e ed/e ed0e ed1e ed2e ed3e ed4efd5d6Zd,ed4efd7d8Z dd1e ed2e ed.e ed/e ed9e!d:e"d3e efd;d<Z#d-e ed.e ed/e ed0e ed1e ed2e ed3e ed:e"d4efd=d>Z$ dd.e ed/e ed?e!d@e!dAe ef dBdCZ%d?e!d0e ed.e ed/e ed1e ed2e ed4efdDdEZ&dFdGZ'   dd?e e!d4e e(dHe e)de e*fdIdJZ+ ddKe e*dHe e)defdLdMZ,     dd?e e!dNe e(d4e e(d,e e-e!efdHe e)def dOdPZ. ddQe!dRe)dNe/dSe efdTdUZ0dVe!dNe/fdWdXZ1ddYe(dZed[e(de(fd\d]Z2d^e!deee effd_d`Z3d^e!fdadbZ4dce!d9e!dde edAe edee edfe ef dgdhZ5 idd^e!d4edje"fdkdlZ6dmedne"doedpe edqef drdsZ7e8dHe)defdtduZ9dved?e!dRe)fdwdxZ:dved?e!dRe)fdydzZ;d{e*ddedAe edee ed|ef d}d~ZddedAe edee ed|efddZ?ddedAedeed|edef ddZ@defddZAdede"de"fddZBde e"de e"de"fddZCdeDdeEeFee(e e*ffdeDee(geEdfdeGdfddZHddZIddZJddZKddZLddZMddZNdee-ee/ffddZOdeePfddZQdeeRfddZSd1e ed2e ede e"de e"d:e"f ddZTdedede e"de e"d:e"dePf ddZUdePfddZVdNe/fddZWd.e ed/e ed:e"de e"de e"f ddZXd.ed/ede e"de e"d:e"de/f ddZYd3e ede e"de e"d:e"fddńZZd3ede e"de e"d:e"deRf ddDŽZ[deRfddʄZ\de]de"fdd̈́Z^de(de(de e"fddфZ_e8de`fddԄZae8ddքZbdS)PrometheusLoggerc KszCddlm}m}m}||_|||_|||_|||_ |jdd| dd|_ |jdd| dd|_ |j dd | dt d |_|j d d | d t d |_|j d d| d t d |_|jdd| dd|_|jdd| dd|_|jdd| dd|_|jdd| dd|_|jdd| dd|_|jdd| dd|_|jdd| dd|_|jdd| dd|_|jd d!| d d|_|jd"d#| d"d|_|jd$d%| d$d|_|jd&d'| d&d|_|jd(d)| d(d|_|jd*d+| d*d|_|jd,d-| d,d|_|jd.d/| d.d|_ |jd0d1| d0d|_!|j d2d3| d2t d |_"|j d4d5| d4t d |_#|j d6d7gd8t d |_$|jd9d:gd;d|_%|jdd|_&|jd?d@dAgd|_'|jdBdC| dBd|_(|jdDdE| dDd|_)|jdFdG| dFd|_*|jdHdI| dHd|_+|jdJdK| dJd|_,|jdLdM| dLd|_-|jdNdO| dNd|_.|j dPdQ| dPd|_/|dRdS| dR|_0|dTdU| dT|_1|jdVdWdXgd|_2|jdYdZ| dYd|_3|jd[d\| d[d|_4|jd]d^| d]d|_5|jd_d`| d_d|_6|jdadb| dad|_7|jdcddgd|_8|jdedfgd|_9WdSt:y\}z t;dgt<||d}~ww)hNr)CounterGauge Histogram$litellm_proxy_failed_requests_metriczjTotal number of failed responses from proxy - the client did not get a success response from litellm proxy)nameZ documentation labelnames#litellm_proxy_total_requests_metriczXTotal number of requests made to the proxy server - track number of client side requests$litellm_request_total_latency_metricz0Total latency (seconds) for a request to LiteLLM)r+Zbucketslitellm_llm_api_latency_metricz1Total latency (seconds) for a models LLM API call*litellm_llm_api_time_to_first_token_metricz-Time to first token for a models LLM API calllitellm_spend_metriczTotal spend on LLM requests)r+litellm_total_tokens_metricz7Total number of input + output tokens from LLM requestslitellm_input_tokens_metricz.Total number of input tokens from LLM requestslitellm_output_tokens_metricz/Total number of output tokens from LLM requests$litellm_remaining_team_budget_metriczRemaining budget for teamlitellm_team_max_budget_metriczMaximum budget set for team*litellm_team_budget_remaining_hours_metricz*Remaining days for team budget to be reset'litellm_remaining_api_key_budget_metriczRemaining budget for api key!litellm_api_key_max_budget_metriczMaximum budget set for api key-litellm_api_key_budget_remaining_hours_metricz.Remaining hours for api key budget to be reset$litellm_remaining_user_budget_metriczRemaining budget for userlitellm_user_max_budget_metriczMaximum budget set for user*litellm_user_budget_remaining_hours_metricz+Remaining hours for user budget to be reset,litellm_remaining_api_key_requests_for_modelzLRemaining Requests API Key can make for model (model based rpm limit on key)*litellm_remaining_api_key_tokens_for_modelzJRemaining Tokens API Key can make for model (model based tpm limit on key)!litellm_remaining_requests_metriczWLLM Deployment Analytics - remaining requests for model, returned from LLM API Providerlitellm_remaining_tokens_metricz:remaining tokens for model, returned from LLM API Providerlitellm_overhead_latency_metricz;Latency overhead (milliseconds) added by LiteLLM processing"litellm_request_queue_time_secondsz>Time spent in request queue before processing starts (seconds)Z!litellm_guardrail_latency_secondsz)Latency (seconds) for guardrail executionguardrail_namestatus error_type hook_typelitellm_guardrail_errors_totalz=Total number of errors encountered during guardrail executionrDrFrG litellm_guardrail_requests_totalz%Total number of guardrail invocationsrDrErG(litellm_provider_remaining_budget_metriczHRemaining budget for provider - used when you set provider budget limits api_providerlitellm_deployment_statezlLLM Deployment Analytics - The state of the deployment: 0 = healthy, 1 = partial outage, 2 = complete outagelitellm_deployment_tpm_limitz$Deployment TPM limit found in configlitellm_deployment_rpm_limitz$Deployment RPM limit found in configlitellm_deployment_cooled_downzLLM Deployment Analytics - Number of times a deployment has been cooled down by LiteLLM load balancing logic. exception_status is the status of the exception that caused the deployment to be cooled down$litellm_deployment_success_responseszOLLM Deployment Analytics - Total number of successful LLM API calls via litellm$litellm_deployment_failure_responseszLLM Deployment Analytics - Total number of failed LLM API calls for a specific LLM deploymeny. exception_status is the status of the exception from the llm api!litellm_deployment_total_requestszXLLM Deployment Analytics - Total number of LLM API calls via litellm - success + failure+litellm_deployment_latency_per_output_tokenz3LLM Deployment Analytics - Latency per output token'litellm_deployment_successful_fallbackszfLLM Deployment Analytics - Number of successful fallback requests from primary model -> fallback model#litellm_deployment_failed_fallbackszbLLM Deployment Analytics - Number of failed fallback requests from primary model -> fallback model(litellm_callback_logging_failures_metriczTTotal number of failures when emitting logs to callbacks (e.g. s3_v2, langfuse, etc) callback_name&litellm_llm_api_failed_requests_metricz5deprecated - use litellm_proxy_failed_requests_metriclitellm_requests_metriczdeprecated - use litellm_proxy_total_requests_metric. Total number of LLM calls to litellm - track total per API Key, team, userlitellm_cache_hits_metricz"Total number of LiteLLM cache hitslitellm_cache_misses_metricz$Total number of LiteLLM cache misseslitellm_cached_tokens_metricz&Total tokens served from LiteLLM cacheZlitellm_total_usersz Total number of users in LiteLLMZlitellm_teams_countz Total number of teams in LiteLLMz(Got exception on init prometheus client )=prometheus_clientr&r'r(_parse_prometheus_config label_filters_create_metric_factoryZ_counter_factoryZ_gauge_factoryZ_histogram_factoryget_labels_for_metricr)r,ZLATENCY_BUCKETSr-r.r/r0litellm_tokens_metricr2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rA!litellm_request_queue_time_metric litellm_guardrail_latency_metricrHrJrLrNrOrPrQrRrSrTrUrVrWrXrZr[r\r]r^litellm_total_users_metriclitellm_teams_count_metric Exceptionrstr)selfkwargsr&r'r(er"r"r#__init__DsF                 zPrometheusLogger.__init__returnc Csddl}ddlm}|j}|siStd|g}t|_|D]}t|t r0|di|}n|}| ||j |j q!| |}|jrX||dd|j}t|||} || | S)zNParse prometheus metrics configuration for label filtering and enabled metricsrN)PrometheusMetricsConfigzprometheus config: z!Configuration validation failed:  r")litellm%litellm.types.integrations.prometheusrpZprometheus_metrics_configrdebugsetenabled_metrics isinstancedictappendupdatemetrics_validate_all_configurationsZ has_errors_pretty_print_validation_errorsjoinZall_error_messages ValueError_build_label_filters_pretty_print_prometheus_config) rkrrrpconfigparsed_configsZ group_configZ parsed_configvalidation_results error_messagerar"r"r#r`s0       z)PrometheusLogger._parse_prometheus_configrcCsfg}g}|D]&}|jD] }||}|r||q |jr+|||j}|r+||q qt||dS)z>Validate all metric configurations and return collected errors) metric_errors label_errors)r{_validate_single_metric_nameryinclude_labels_validate_single_metric_labelsValidationResults)rkrrrr metric_name metric_error label_errorr"r"r#r|s"     z-PrometheusLogger._validate_all_configurationsrcCs0ddlm}|t|tvrt||tdSdS)zValidate a single metric namer)get_args)r valid_metricsN)typingrruDEFINED_PROMETHEUS_METRICSMetricValidationError)rkrrr"r"r#rs z-PrometheusLogger._validate_single_metric_namelabelscsDddlm}t|t|fdd|D}|r t||dSdS)z#Validate labels for a single metricr)rcsg|]}|vr|qSr"r".0label valid_labelsr"r# szCPrometheusLogger._validate_single_metric_labels..rinvalid_labelsrN)rrPrometheusMetricLabels get_labelsrLabelValidationError)rkrrrrr"rr#rs z/PrometheusLogger._validate_single_metric_labelscCs<i}|D]}|jD]}|jr||dur|j||<q q|S)z1Build label filters from validated configurationsN)r{rr)rkrrarrr"r"r#r%s  z%PrometheusLogger._build_label_filterscCs4|||}|r|j|j|j|jdt|jdS)z Ensure that all the configured labels are valid for the metric Raises ValueError if the metric labels are invalid and pretty prints the error rT)r"_pretty_print_invalid_labels_errorrrrrmessage)rkrrrr"r"r#"_validate_configured_metric_labels2s  z3PrometheusLogger._validate_configured_metric_labelsrNcCszddlm}ddlm}ddlm}ddlm}|}|ddd}|d |||d d |j rxd d |j D}|j dj } |dd |dd} ||| d d |dddddd} | j ddddt | D]} | | qk|| |jr|jD]D} |d| jdd | jdd}|||d d |d| jdddddd}|j ddddt | jD]}||q||q~|d WdSty|j D]}t|jq|jD]}t|jqYdSw)z-Pretty print all validation errors using richrConsolePanelTableTextu(🚨🚨 Configuration Validation Errorsbold redstylerqred border_stylecSsg|]}|jqSr"r)rrmr"r"r#r\szDPrometheusLogger._pretty_print_validation_errors..zInvalid Metric Names: , 📊 Valid Metric NamesT bold greenleftgreentitle show_header header_style title_justifyrAvailable Metricscyanrno_wrapzInvalid Labels for '': u🏷️ Valid Labels for '' Valid LabelsN) rich.consoler rich.panelr rich.tabler rich.textrprintrrr~ add_columnsortedadd_rowrrrr ImportErrorrerrorr)rkrrrrrconsolerZinvalid_metricsrZmetrics_error_text metrics_tablemetricrZlabels_error_text labels_tablerrrr"r"r#r}Gs|                z0PrometheusLogger._pretty_print_validation_errorsrrc Cszcddlm}ddlm}ddlm}ddlm}|}|d|dd|d d d } |d d dddd} | j ddd dt |D]} | | qA| d| || dd| | | dWdSt y}td|d|dt |YdSw)z8Pretty print error message for invalid labels using richrrrrru%🚨🚨 Invalid Labels for Metric: 'z' Invalid labels: rz' Please specify only valid labels belowrru$🏷️ Valid Labels for this MetricTrrrrrrrrqrrzInvalid labels for metric 'rz. Valid labels: N)rrrrrrrrr~rrrrrrr) rkrrrrrrrrrrrr"r"r#rs:          z3PrometheusLogger._pretty_print_invalid_labels_errorinvalid_metric_namerc Csz]ddlm}ddlm}ddlm}ddlm}|}|d|ddd }|d d d d dd} | jddd dt |D]} | | q;| d| ||dd| | | dWdSt ytt d|dt |YdSw)z=Pretty print error message for invalid metric name using richrrrrru🚨🚨 Invalid Metric Name: 'z1' Please specify one of the allowed metrics belowrrrTrrrrrrrrqrrzInvalid metric name: z. Valid metrics: N)rrrrrrrrrrrrrrr) rkrrrrrrrrrrr"r"r#"_pretty_print_invalid_metric_errors:           z3PrometheusLogger._pretty_print_invalid_metric_errorcCs.||}|r|j|j|jdt|jdS)z] Raises ValueError if the metric name is invalid and pretty prints the error )rrN)rrrrrr)rkrrr"r"r#_valid_metric_names  z#PrometheusLogger._valid_metric_namerac Cszddlm}ddlm}ddlm}ddlm}|}|ddd}|d d d d d }|jddd dt |drH|j rHt |j D]} | | q?n| d|dd dd d } | jddd d| jddd|rt | D]\} } | rwd| nd} | | | qln| dd|d|||dd|||| |dWd!Stytdt |drt |j ndtd |Yd!Sw)"z>Pretty print the processed prometheus configuration using richrrrrrz"Prometheus Configuration Processedz bold blueru📊 Enabled MetricsTz bold magentar)rrrrz Metric Namerrrvz8[yellow]All metrics enabled (no filter applied)[/yellow]u🏷️ Label FiltersrzAllowed Labelsyellowrz[dim]No labels specified[/dim]z+[yellow]No label filtering applied[/yellow]z*[dim]All default labels will be used[/dim]rqbluerzEnabled metrics: z All metricszLabel filters: N)rrrrrrrrrhasattrrvrritemsr~rrrinfo)rkrarrrrrrrrrrrZ labels_strr"r"r#rsh           z0PrometheusLogger._pretty_print_prometheus_configcCs"t|dsdS|js dS||jvS)z3Check if a metric is enabled based on configurationrvT)rrv)rkrr"r"r#_is_metric_enabledDs  z#PrometheusLogger._is_metric_enabledcsfdd}|S)zMCreate a factory function that returns either a real metric or a no-op metriccs6|r|dn|dd}|r|i|StS)Nrr*)getrZ NoOpMetric)argsrlr metric_classrkr"r#factorySs z8PrometheusLogger._create_metric_factory..factoryr")rkrrr"rr#rbPs z'PrometheusLogger._create_metric_factorycs8t|}||jvr |S|j|fdd|D}|S)zE Get the labels for a metric, filtered if configured csg|]}|vr|qSr"r"rZconfigured_labelsr"r#rosz:PrometheusLogger.get_labels_for_metric..)rrra)rkrZdefault_labelsfiltered_labelsr"rr#rc^s    z&PrometheusLogger.get_labels_for_metricc sZddlm}td||d}|dust|ts$td||j||dr-dS|dd}|d ip:i}|d pAi} t } | |d d } |d d } |d d} |d d}|d d}|d d}|d}|d}|d}|d d}|d d}i|r|ni|r|ni}|durt|tr|d}ng}t d|d|d|d| d| t dDid| d| d|d |d!d!|d!d"|d#|d$| d%|d d&d'd(d|d)|d*|d+|d+d,|d,d-|d.d/dd0dd1t |d2d3|d d4d5|d d6d7|d d7d8t jr%t|d8nd}| durDt| trD| d9rDdd:lm}|| } |j| | ||||| ||d; |j|| | ||||| |d< |j||| |||| d=IdH|j| ||| |jd>|j||| ||||d?|||||||j||d@t|jdAdB|dC}|jjdDi|dS)ENrrz@prometheus Logging - Enters success logging function for kwargs standard_logging_objectz)standard_logging_object is required, got=rlstandard_logging_payloadmodelrlitellm_paramsmetadata prometheus service_typeuser_api_key_user_iduser_api_key_hashuser_api_key_aliasuser_api_key_team_iduser_api_key_team_aliascompletion_tokens total_tokens response_costZrequester_metadatauser_api_key_auth_metadata request_tagsz'inside track_prometheus_metrics, model z, response_cost z, tokens_used z, end_user_id z, user_api_key end_userhashed_api_key api_key_aliasrequested_model model_groupteam team_aliasuser user_emailZuser_api_key_user_email status_codeZ200litellm_model_nametagsmodel_idapi_baserMcustom_llm_providerexception_statusexception_classcustom_metadata_labelsrrouteZuser_api_key_request_route client_iprequester_ip_address user_agentstreamzsk-) hash_token) end_user_id user_api_keyrr user_api_teamuser_api_team_aliasuser_idr enum_values) rrrrrrrrr)rrrrrrr)rrrlrr)rlrrrrrr)rrr,rsupported_enum_labelsrr") litellm.types.utilsrrrtrrwrxr$_should_skip_metrics_for_invalid_keyr$rUserAPIKeyLabelValuesget_custom_labels_from_metadatarrprometheus_emit_stream_labelrj startswithZlitellm.proxy.utilsr ._increment_top_level_request_and_spend_metrics_increment_token_metrics#_increment_remaining_budget_metrics#_set_virtual_key_rate_limit_metricsr_set_latency_metrics"set_llm_deployment_success_metrics_increment_cache_metricsprometheus_label_factoryrcr,rinc)rkrl response_obj start_timeend_timerrrr _metadatar rrrrrr output_tokensZ tokens_usedrZ_requester_metadatarZcombined_metadata_tagsrr _labelsr"r"r#async_log_success_eventusd                  !      z(PrometheusLogger.async_log_success_eventrrrrrrrrrc Cstd|durt|tr|d} t|jdd| d} |jjd i| |dt|jdd| d} |j jd i| |dt|jd d| d} |j jd i| |d dS) Nz2prometheus Logging - Enters token metrics functionrr1rrrr2Z prompt_tokensr3rr") rrtrwrxr#rcrdrr$r2r3) rkrrrrrrrrrr*r+r"r"r#r4s@ z)PrometheusLogger._increment_token_metricscCs|d}|dur dS|durHt|jdd|d}|jjd i||dd}|dkrFt|jd d|d}|jjd i||dSdSt|jd d|d}|jjd i|dS) z Increment cache-related Prometheus metrics based on cache hit/miss status. Args: standard_logging_payload: Contains cache_hit field (True/False/None) enum_values: Label values for Prometheus metrics cache_hitNTr\rrrrr^r]r")rr#rcr\rr$r^r])rkrrr-r+rr"r"r#r"gs8   z)PrometheusLogger._increment_cache_metricsrrc s|dpi}|dd} |dd} |dd} |dd} |dd} |dd}tj|j|||| | d|j||| | |d |j|| ||d d d IdH}t|D]\}}t|trkt d gd|d|qSdS)NrZuser_api_key_team_spendZuser_api_key_team_max_budgetZuser_api_key_spendZuser_api_key_max_budgetZuser_api_key_user_spendZuser_api_key_user_max_budget)rrrkey_max_budget key_spend)rr team_spendteam_max_budgetr)r user_spenduser_max_budgetrT)return_exceptionsz0[Non-Blocking] Prometheus: Budget metric lookup )keyrrz failed: ) rasynciogather-_set_api_key_budget_metrics_after_api_request*_set_team_budget_metrics_after_api_request*_set_user_budget_metrics_after_api_request enumeraterwrirrt)rkrrrrrrrr(Z _team_spendZ_team_max_budgetZ_api_key_spendZ_api_key_max_budgetZ _user_spendZ_user_max_budgetresultsirr"r"r#rsL        z4PrometheusLogger._increment_remaining_budget_metricsc CsZt|jdd| d} |jjdi| t|jdd| d} |jjdi| |dS)Nr[rrr0r")r#rcr[rr$r0) rkrrrrrrrrrr+r"r"r#rs z?PrometheusLogger._increment_top_level_request_and_spend_metricsrlrrc Csddlm}||}d|}d|} ||tjptj} || tjp'tj} |jt|t|t|t|| |j t|t|t|t|| dS)Nr)#get_model_group_from_litellm_kwargszlitellm-key-remaining-requests-zlitellm-key-remaining-tokens-) Z)litellm.proxy.common_utils.callback_utilsr?rsysmaxsizer=rrrur>) rkrrrlrrr?rZ remaining_requests_variable_nameZremaining_tokens_variable_nameremaining_requestsremaining_tokensr"r"r#rs.   z4PrometheusLogger._set_virtual_key_rate_limit_metricscCs|dpt}|d} |dd} |dd} |j| | d} | durD|dddurDt|jd d |d } |jjdi| | nt d |j| |d}|durjt|jd d |d }|j jdi|||j| |d}|durt|jdd |d }|j jdi|||dipi}|dpid}|dur|dkrt|jdd |d }|j jdi||dSdSdS)Nr'r&api_call_start_timecompletion_start_time)r&r'r FTr/rrzUTime to first token metric not emitted, stream option in model_parameters is not Truer.r-rrqueue_time_secondsrrBr")rrnow_safe_duration_secondsr#rcr/robserverrtr.r-re)rkrlrrrrrrr'r&rDrEZtime_to_first_token_secondsZ _ttft_labelsZapi_call_total_time_secondsr+Ztotal_time_seconds_litellm_paramsrFr"r"r#r s     z%PrometheusLogger._set_latency_metricscs8ddlm}td||di}|j||drdS|dd}|dip+i}t} | |d d } |d d } |d d } |d d} |d d}|d d}z+|jt | t | t | t |t |t |t | t |dd | |WdSt y}zt dt|WYd}~dSd}~ww)Nrrz@prometheus Logging - Enters failure logging function for kwargs rrrrrrrrrrrrrr0prometheus Layer Error(): Exception occured - {})rrrrtrrr$rZrrr$"set_llm_deployment_failure_metricsri exceptionformatrj)rkrlr%r&r'rrrrr rrrrrrrmr"r"r#async_log_failure_eventcs^         z(PrometheusLogger.async_log_failure_eventrMc Csd}|rt|dr|jrzt|j}Wn ttfyYnw|sG|rGt|ddp.t|dd}|durGzt|}Wn ttfyFd}Ynw|sx|rx|d}|rxt|ddp]t|dd}|durxzt|}W|Sttfywd}Y|Sw|S)aT Extract HTTP status code from various input formats for validation. This is a centralized helper to extract status code from different callback function signatures. Handles both ProxyException (uses 'code') and standard exceptions (uses 'status_code'). Args: kwargs: Dictionary potentially containing 'exception' key enum_values: Object with 'status_code' attribute exception: Exception object to extract status code from directly Returns: Status code as integer if found, None otherwise NrcoderM)rrintr TypeErrorgetattrr)rkrlrrMrZexception_in_kwargsr"r"r#_extract_status_codesN    z%PrometheusLogger._extract_status_codercsF|dkrdS|dur!t|gd}tfdd|Dr!dSdS)a Determine if a request has an invalid API key based on status code and exception. This method prevents invalid authentication attempts from being recorded in Prometheus metrics. A 401 status code is the definitive indicator of authentication failure. Additionally, we check exception messages for authentication error patterns to catch cases where the exception hasn't been converted to a ProxyException yet. Args: status_code: HTTP status code (401 indicates authentication error) exception: Exception object to check for auth-related error messages Returns: True if the request has an invalid API key and metrics should be skipped, False otherwise iTN)zvirtual key expectedzexpected to start with 'sk-'zauthentication errorzinvalid api keyzapi key not validc3s|]}|vVqdSNr")rpatternZ exception_strr"r# sz?PrometheusLogger._is_invalid_api_key_request..F)rjlowerany)rkrrMZauth_error_patternsr"rWr#_is_invalid_api_key_requests z,PrometheusLogger._is_invalid_api_key_requestuser_api_key_dictcCs`|j|||d}|dur|r|d}|j||dr.td|d|r't|jnddSdS) aD Determine if Prometheus metrics should be skipped for invalid API key requests. This is a centralized validation method that extracts status code and exception information from various callback function signatures and determines if the request represents an invalid API key attempt that should be filtered from metrics. Args: kwargs: Dictionary potentially containing exception and other data user_api_key_dict: User API key authentication object (currently unused) enum_values: Object with status_code attribute standard_logging_payload: Standard logging payload dictionary exception: Exception object to check directly Returns: True if metrics should be skipped (invalid key detected), False otherwise )rlrrMNrMrMzESkipping Prometheus metrics for invalid API key request: status_code=z , exception=TF)rTrr[rrttype__name__)rkrlr\rrrMrr"r"r#rs"  z5PrometheusLogger._should_skip_metrics_for_invalid_key request_dataoriginal_exception traceback_strc s$ddlm}|j||drdS|j|d}z|j||did}|dip)i}|d id p;|d id } td$id |jd |jd |j d|j d|j d|j d|j d|dddt|dt|d||d|d|jd|dd|dd| dtjrt|dn1d} t|jdd | d!} |jjd$i| t|jd"d | d!} |jjd$i| WdS} t|jdd | d!} |jjd$i| t|jd"d | d!} |jjd$i| WdSty} ztd#t| WYd} ~ dSd} ~ ww)%ar Track client side failures Proxy level tracking - failed client side requests labelnames=[ "end_user", "hashed_api_key", "api_key_alias", REQUESTED_MODEL, "team", "team_alias", ] + EXCEPTION_LABELS, r)StandardLoggingPayloadSetup)r\rMNr]proxy_server_request)rrdr model_infoidrrrrrrrrrrrrrrrr r r rr r)rrr,rKr")*litellm.litellm_core_utils.litellm_loggingrcrrTZ_get_request_tagsrrrrrapi_key key_aliasteam_idrrj_get_exception_class_nameZ request_routerrrr#rcr)rr$r,rirrMrN) rkr`rar\rbrcrr*r(rrr+rmr"r"r#async_post_call_failure_hook#s            z-PrometheusLogger.async_post_call_failure_hookdatacsdS)a^ Proxy level tracking - triggered when the proxy responds with a success response to the client Note: litellm_proxy_total_requests_metric is NOT incremented here to avoid double-counting. It is incremented in async_log_success_event which fires for all successful requests (both streaming and non-streaming). Nr")rkrmr\responser"r"r#async_post_call_success_hookxs z-PrometheusLogger.async_post_call_success_hookobjr5defaultcCs.|dur|St|tr|||St|||S)z&Get value from dict or Pydantic model.N)rwrxrrS)rkrpr5rqr"r"r# _safe_gets    zPrometheusLogger._safe_getrequest_kwargsc s>|dipi}|dipi}||dpi}t|tr |n!t|ddt|ddt|ddt|ddt|d dt|d dd |dipHid d ttffdd }d ttffdd }d ttffdd }d ttffdd }||||d pd d pd dS)z Extract label values for deployment failure metrics from all available sources in request_kwargs. Falls back to litellm_params metadata and user_api_key_auth when standard_logging_payload has None values. rrrrNrrrr r )rrrrr r user_api_key_authrocDd}|dur |Sd}|dur|Sdur tddSdS)NrrirrSvalZ_litellm_params_metadatar(rtr"r#_get_api_key_alias   zUPrometheusLogger._extract_deployment_failure_label_values.._get_api_key_aliascru)Nrrjrvrwryr"r# _get_team_idr{zOPrometheusLogger._extract_deployment_failure_label_values.._get_team_idcru)Nrrrvrwryr"r#_get_team_aliasr{zRPrometheusLogger._extract_deployment_failure_label_values.._get_team_aliascsPd}|dur |Sd}|dur|Sdur&tddp%tddSdS)NrrhZ api_key_hashrvrwryr"r#_get_hashed_api_keys  zVPrometheusLogger._extract_deployment_failure_label_values.._get_hashed_api_key)rrrrr r )rrrrwrxrSr rj) rkrsrrJZ _metadata_rawrzr|r}r~r"ryr#(_extract_deployment_failure_label_valuessH          z9PrometheusLogger._extract_deployment_failure_label_valuescCszDtd|di}|dipi}|dd}|dd}|dd}|dd}|d d}|durR|d p=id pL|d pGid pLi} | d }|durj|d p\idpi|d pfid}|dd} |j||drzWdS||} |d ipi} | dp| d} | dp| d}| dp| d}| dp| d}| dp| d}| dp| d}d}|durtt|dd}|dks|st|dd}|durt|}t|||| ||r||nd|p|| ||||dg||d} |j |p d ||| pd d!|dur.t |j d"d#|d$}|j j d'i|t |j d%d#|d$}|jj d'i|WdStyc}ztd&t|WYd}~dSd}~ww)(a Sets Failure metrics when an LLM API call fails - mark the deployment as partial outage - increment deployment failure responses metric - increment deployment total requests metric Args: request_kwargs: dict (setting remaining tokens requests metricrrrNrrrrMZlitellm_metadatarerrfrrrrrrrrrrr r r rNonerPr)rrrrMrrrrrrrrr r rrrrrMrSrrrTzLPrometheus Error: set_llm_deployment_failure_metrics. Exception occured - {}r")rrtrrrrjrSrrkset_deployment_partial_outager#rcrSrr$rTrirN)rkrsrrJrrrrrM _model_info llm_providerZfallback_valuesr(rrrrr r rrPrr+rmr"r"r#rLs            z3PrometheusLogger.set_llm_deployment_failure_metricsrerrrc Cs|dp |d}|dp|d}|dur4t|jddt||||dd} |jjd i| ||durVt|jddt||||dd} |jjd i| |dSdS) z? Set the deployment TPM and RPM limits metrics tpmrpmNrOrrrrPr")rr#rcrrOrrurP) rkrerrrrrrrr+r"r"r#%_set_deployment_tpm_rpm_limit_metricsos:   z6PrometheusLogger._set_deployment_tpm_rpm_limit_metrics?r)c CszPtd|d}|durWdS|j|||drWdS|d}|dip)i}t|} |dd} |dd} | dp@i} | d d} | sK|rV|j| || | || d d}d}|d d }rn|d d}|dd}|d d}rt|jdd|d}|jj d i| |d|r t|jdd|d}|j j d i| ||rt|jdd|d}|j j d i| | |j| pd| pd|pd| pddt|jdd|d}|jj d i|t|jdd|d}|jj d i|||}d}|dddur|ddur|d||}|p|}|}d}|durL|dkrO||}t|jdd|d}|jj d i| |WdSWdSWdStyo}ztdt|WYd}~dSd}~ww)!Nrr)rlrrrrrrrerf)rerrrrrZ hidden_paramsadditional_headersZx_ratelimit_remaining_requestsZx_ratelimit_remaining_tokenslitellm_overhead_time_msrArrir?r@rrrRrTr TrErrUzLPrometheus Error: set_llm_deployment_success_metrics. Exception occured - {}r")rrtrrrrr#rcrArrIr?rur@set_deployment_healthyrRr$rT total_secondsrUrirMrNrj)rkrsr&r'rr)rrrJr(rrrrrBrCrrr+Z response_msZ!time_to_first_token_response_timeZ_latencyZ_latency_secondsZlatency_per_tokenrmr"r"r#r!s         z3PrometheusLogger.set_llm_deployment_success_metricsrDlatency_secondsrErFrGc Csz4|jj|||p d|d||jj|||d|dkr/|r2|jj|||dWdSWdSWdStyQ}ztdt |WYd}~dSd}~ww)a\ Record guardrail metrics for prometheus. Args: guardrail_name: Name of the guardrail latency_seconds: Execution latency in seconds status: "success" or "error" error_type: Type of error if any, None otherwise hook_type: "pre_call", "during_call", or "post_call" nonerCrKrrIz#Error recording guardrail metrics: N) rfrrIrJr$rHrirrtrj)rkrDrrErFrGrmr"r"r#_record_guardrail_metrics?s4   "z*PrometheusLogger._record_guardrail_metricscCsTd}t|drt|dp d}t|dkr"|d|ddd}||jj7}|S)Nrrr.)rrSlenupper __class__r_)rMZexception_class_namer"r"r#rkks   z*PrometheusLogger._get_exception_class_nameoriginal_model_groupc sddlm}m}td||t|}||pi}|j|d}|d} tt t |dp0g} t || |d|d|d |d t t |d d | || d } t|jdd| d} |jjdi| d S)zF Logs a successful LLM fallback event on prometheus rStandardLoggingMetadatarczLPrometheus: log_success_fallback_event, original_model_group: %s, kwargs: %srrrrrrrrN rZfallback_modelrrrrrrrrVrrr")rgrrcrrtrrget_standard_logging_metadatarr rjrrSrkr#rcrVrr$) rkrrlrarrc _metadata_keyr(standard_metadata _new_modelr*rr+r"r"r#log_success_fallback_event{sB  z+PrometheusLogger.log_success_fallback_eventc sddlm}m}td|||d}t|}||pi}ttt |dp*g} |j |d} t ||| d| d| d | d t t |d d | || d } t|jdd| d} |jjdi| d S)z@ Logs a failed LLM fallback event on prometheus rrzLPrometheus: log_failure_fallback_event, original_model_group: %s, kwargs: %srrrrrrrrNrrWrrr")rgrrcrrtrrrr rjrrrSrkr#rcrWrr$) rkrrlrarrcrrr(r*rrr+r"r"r#log_failure_fallback_eventsB  z+PrometheusLogger.log_failure_fallback_eventstaterMcCs<t|jddt||||dd}|jjdi||dS)z+ Set the deployment state. rNrrrNr")r#rcrrNrru)rkrrrrrMr+r"r"r#set_litellm_deployment_states  z-PrometheusLogger.set_litellm_deployment_statecC|d||||dS)NrrrkrrrrMr"r"r#r z'PrometheusLogger.set_deployment_healthycCr)Nrrrr"r"r#rrz.PrometheusLogger.set_deployment_partial_outagecCr)Nrrr"r"r#set_deployment_complete_outage rz/PrometheusLogger.set_deployment_complete_outagercCs0|jt|t|t|t|t|dS)zn increment metric when litellm.Router / load balancing logic places a deployment in cool down N)rQrrr$)rkrrrrMrr"r"r# increment_deployment_cooled_down s  z1PrometheusLogger.increment_deployment_cooled_downrYcCs|jj|ddS)za Increment metric when logging to a callback fails (e.g., s3_v2, langfuse, etc.) )rYN)rXrr$)rkrYr"r"r#"increment_callback_logging_failure% s z3PrometheusLogger.increment_callback_logging_failureproviderspend budget_limitcCs |j||j||ddS)z? Track provider remaining budget in Prometheus  max_budgetrN)rLrru_safe_get_remaining_budget)rkrrrr"r"r#track_provider_remaining_budget0 s z0PrometheusLogger.track_provider_remaining_budgetrcCs$|durtdS|dur|S||S)Ninf)float)rkrrr"r"r#r= s z+PrometheusLogger._safe_get_remaining_budgetdata_fetch_function.set_metrics_function data_type)teamskeysusersc sddlm}|dur dSzEd}d}|||dIdH\}}|dur%t|}||d|} ||IdHtd| dD]}|||dIdH\}} ||IdHq;WdStyr} ztd|d t| WYd} ~ dSd} ~ ww) ab Generic method to initialize budget metrics for teams or API keys. Args: data_fetch_function: Function to fetch data with pagination. set_metrics_function: Function to set metrics for the fetched data. data_type: String representing the type of data ("teams" or "keys") for logging purposes. r prisma_clientNr2) page_sizepagerzError initializing z budget metrics: )litellm.proxy.proxy_serverrrrangerirrMrj) rkrrrrrrrm total_countZ total_pages_rmr"r"r#_initialize_budget_metricsH s0 z+PrometheusLogger._initialize_budget_metricsc sxddlmddlmdurtddSdtdtdttt t tfffd d }|j ||j d d IdHdS) zY Initialize team budget metrics by reusing the generic pagination logic. r)get_paginated_teamsrNzDPrometheus: skipping team metrics initialization, DB not initializedrrrocs2||dIdH\}}|durt|}||fS)N)rrr)r)rrrrrrr"r# fetch_teams szEPrometheusLogger._initialize_team_budget_metrics..fetch_teamsrrrr) Z1litellm.proxy.management_endpoints.team_endpointsrrrrrtrQr r rr r_set_team_list_budget_metrics)rkrr"rr#_initialize_team_budget_metricsu s(   z0PrometheusLogger._initialize_team_budget_metricsc sddlm}ddlmddlmddlmdur$t ddSdt d t d t t |t ttftt fffd d }|j||jd dIdHdS)z\ Initialize API key budget metrics by reusing the generic pagination logic. r)r )UI_SESSION_TOKEN_TEAM_ID)_list_key_helperrNzCPrometheus: skipping key metrics initialization, DB not initializedrrroc sR||ddddddd IdH}|dg}|d}|dur%t|}||fS)NT) rrsizerrjriZkey_hashZexclude_team_idZreturn_full_objectZorganization_idrr)rr)rrZkey_list_responserrrrrr"r# fetch_keys s$ zGPrometheusLogger._initialize_api_key_budget_metrics..fetch_keysrr)rr litellm.constantsrZ;litellm.proxy.management_endpoints.key_management_endpointsrrrrrtrQr r rjrrr r_set_key_list_budget_metrics)rkr rr"rr#"_initialize_api_key_budget_metrics s4    z3PrometheusLogger._initialize_api_key_budget_metricsc svddlm}ddlmdurtddSdtdtdtt|t tfffd d }|j ||j d d IdHdS) zY Initialize user budget metrics by reusing the generic pagination logic. r)rrNzDPrometheus: skipping user metrics initialization, DB not initializedrrrocsF|d|}jjj||ddidIdH}jjIdH}||fS)NrZ created_atZdesc)skipZtakeorder)dblitellm_usertableZ find_manycount)rrrrrrr"r# fetch_users s  zEPrometheusLogger._initialize_user_budget_metrics..fetch_usersrr) litellm.proxy._typesrrrrrtrQr r r r_set_user_list_budget_metrics)rkrrr"rr#_initialize_user_budget_metrics s(   z0PrometheusLogger._initialize_user_budget_metricsc sddlm}ddlm}|jj}|r?|jr?|j|dIdHr=z|IdHW|j |dIdHdS|j |dIdHwdS|IdHdS)a Handler for initializing remaining budget metrics for all teams to avoid metric discrepancies. Runs when prometheus logger starts up. - If redis cache is available, we use the pod lock manager to acquire a lock and initialize the metrics. - Ensures only one pod emits the metrics at a time. - If redis cache is not available, we initialize the metrics directly. r)'PROMETHEUS_EMIT_BUDGET_METRICS_JOB_NAME)proxy_logging_obj)Z cronjob_idN) rrrrZdb_spend_update_writerpod_lock_manager redis_cache acquire_lock$_initialize_remaining_budget_metrics release_lock)rkrrrr"r"r##initialize_remaining_budget_metrics s"    z4PrometheusLogger.initialize_remaining_budget_metricscsHtd|IdH|IdH|IdH|IdHdS)zc Helper to initialize remaining budget metrics for all teams, API keys, and users. z+Emitting key, team, user budget metrics....N)rrtrrr'_initialize_user_and_team_count_metrics)rkr"r"r#r s  z5PrometheusLogger._initialize_remaining_budget_metricsc sddlm}|durtddSz1|jjIdH}|j|td||jj IdH}|j |td|WdSt y`}zt dt |WYd}~dSd}~ww)z Initialize user and team count metrics by querying the database. Updates: - litellm_total_users: Total count of users in the database - litellm_teams_count: Total count of teams in the database rrNzOPrometheus: skipping user/team count metrics initialization, DB not initializedz'Prometheus: set litellm_total_users to z'Prometheus: set litellm_teams_count to z,Error initializing user/team count metrics: )rrrrtrrrrgruZlitellm_teamtablerhrirMrj)rkrZ total_usersZ total_teamsrmr"r"r#r s0     z8PrometheusLogger._initialize_user_and_team_count_metricsrcs$|D] }t|tr||qdS)z8Helper function to set budget metrics for a list of keysN)rwr_set_key_budget_metrics)rkrr5r"r"r#r. s   z-PrometheusLogger._set_key_list_budget_metricsrc|D]}||qdS)z9Helper function to set budget metrics for a list of teamsN)_set_team_budget_metrics)rkrrr"r"r#r6  z.PrometheusLogger._set_team_list_budget_metricsrcr)z9Helper function to set budget metrics for a list of usersN)_set_user_budget_metrics)rkrrr"r"r#r; rz.PrometheusLogger._set_user_list_budget_metricsr0r1cs6|r|j||p d|||dIdH}||dSdS)z Set team budget metrics after an LLM API request - Assemble a LiteLLM_TeamTable object - looks up team info from db if not available in metadata - Set team budget metrics r)rjrrrrN)_assemble_team_objectr)rkrrr0r1r team_objectr"r"r#r9@ s z;PrometheusLogger._set_team_budget_metrics_after_api_requestrjrc sddlm}ddlm}m}|pd|} t||| |d} z ||||dIdH} WntyG} ztdt | | WYd} ~ Sd} ~ ww| r\| j | _ | j dur\| j dur\| j | _ | S)z Assemble a LiteLLM_TeamTable object for fields not available in metadata, we fetch from db Fields not available in metadata: - `budget_reset_at` r)get_team_objectruser_api_key_cache)rjrrr)rjrrNz4[Non-Blocking] Prometheus: Error getting team info: ) litellm.proxy.auth.auth_checksrrrrrrirrtrjbudget_reset_atr) rkrjrrrrrrrZ_total_team_spendrZ team_informr"r"r#rZ s8   z&PrometheusLogger._assemble_team_objectrcCst|j|jpdd}t|jdd|d}|jjd i||j|j |j d|j durCt|jdd|d}|j jd i||j |j duret|jd d|d}|j jd i||j|j d dSdS) z Set team budget metrics for a single team - Remaining Budget - Max Budget - Budget Reset At r)rrr4rrrNr5r6rr")rrjrr#rcr4rrurrrr5rr6%_get_remaining_hours_for_budget_reset)rkrrr+r"r"r#r sJ   z)PrometheusLogger._set_team_budget_metricscCst|j|jpdd}t|jdd|d}|jjd i||j|j |j d|j durCt|jdd|d}|j jd i||j |j dur[|j jd i||j|j d dSdS) z{ Set virtual key budget metrics - Remaining Budget - Max Budget - Budget Reset At r)rrr7rrrNr8rr")rtokenrir#rcr7rrurrrr8rr9r)rkr\rr+r"r"r#r sB  z(PrometheusLogger._set_key_budget_metricsr.r/cs6|r|j||p d|||dIdH}||dSdS)Nr)rrr.r/r)_assemble_key_objectr)rkrrrr.r/r\r"r"r#r8 s z>PrometheusLogger._set_api_key_budget_metrics_after_api_requestc sddlm}ddlm}m}|pd|} t|||| d} z| jr5|| j||dIdH} | r8| j| _W| SW| SW| StyW} zt dt | WYd} ~ | Sd} ~ ww)z2 Assemble a UserAPIKeyAuth object r)get_key_objectr)rrirr)Z hashed_tokenrrNz3[Non-Blocking] Prometheus: Error getting key info: ) rrrrrrrrrirrtrj) rkrrr.r/rrrrZ_total_key_spendr\Z key_objectrmr"r"r#r s>      z%PrometheusLogger._assemble_key_objectr2r3cs0|r|j||||dIdH}||dSdS)z Set user budget metrics after an LLM API request - Assemble a LiteLLM_UserTable object - looks up user info from db if not available in metadata - Set user budget metrics )rrrrN)_assemble_user_objectr)rkrr2r3r user_objectr"r"r#r: s z;PrometheusLogger._set_user_budget_metrics_after_api_requestc sddlm}ddlm}m}|pd|}t|||d} z||||dddIdH} WntyH} ztdt | | WYd} ~ Sd} ~ ww| r]| j | _ | j dur]| j dur]| j | _ | S) z Assemble a LiteLLM_UserTable object for fields not available in metadata, we fetch from db Fields not available in metadata: - `budget_reset_at` r)get_user_objectr)rrrF)rrrZuser_id_upsertZ check_db_onlyNz4[Non-Blocking] Prometheus: Error getting user info: ) rrrrrrrirrtrjrr) rkrrrrrrrZ_total_user_spendrZ user_informr"r"r#r3 s:   z&PrometheusLogger._assemble_user_objectrcCst|jd}t|jdd|d}|jjd i||j|j|j d|jdur?t|jdd|d}|j jd i||j|j durat|jdd|d}|j jd i||j |j d dSdS) z Set user budget metrics for a single user - Remaining Budget - Max Budget - Budget Reset At )rr:rrrNr;r<rr")rrr#rcr:rrurrrr;rr<r)rkrrr+r"r"r#ra sH   z)PrometheusLogger._set_user_budget_metricsrcCs|t|jdS)z6 Get remaining hours for budget reset i)rrGtzinfor)rkrr"r"r#r sz6PrometheusLogger._get_remaining_hours_for_budget_resetr&r'cCs$t|trt|tr||SdS)z Compute the duration in seconds between two objects. Returns the duration as a float if both start and end are instances of datetime, otherwise returns None. N)rwrr)rkr&r'r"r"r#rH s z'PrometheusLogger._safe_duration_seconds schedulercCs|ddlm}ddlm}tjjtd}t dt |t |dkr ) 0 U   ^ 1 & , / .               -0 #   , 3+  %  . 2 r%rrtagroc s|}fdd|D}tjj|vr#t}|d|jidd|d<|jdur@|jD]\}}t|}|vr?t |||<q-|j dur]t |j } | D]\}}|vr\t |||<qND] } | |vrid|| <q_|S)z Returns a dictionary of label + values for prometheus. Ensures end_user param is not sent to prometheus if it is not supported. cs"i|] \}}|vr|t|qSr")r)rrvaluerr"r# s z,prometheus_label_factory..Zuser_api_key_end_user_idr)rrrN) Z model_dumprZUserAPIKeyLabelNamesZEND_USERrr$rrrrrget_custom_labels_from_tags) rrr enum_dictrr r5rZ sanitized_keyZcustom_tag_labelsrr"rr#r# s6       r#rcCstj}|dus t|dkriSi}|D]C}|}|dr#|dddn|}|d}|}|D]}t|tr<||d}nd}|durDnq.|durVt|t rV|||dd<q|S)z) Get custom labels from metadata Nrz metadata.rrrr) rrZ!custom_prometheus_metadata_labelsrrreplacesplitrwrxrrj)rrresultr5Z original_keyZ keys_partsrpartr"r"r#r s(  rrconfigured_tagcs<ddlddlm}|}||tfdd|DS)a Check if any of the request tags matches a wildcard configured pattern Args: tags: List[str] - The request tags configured_tag: str - The configured tag Returns: bool - True if any of the request tags matches the configured tag, False otherwise e.g. tags = ["User-Agent: curl/7.68.0", "User-Agent: python-requests/2.28.1", "prod"] configured_tag = "User-Agent: curl/*" _tag_matches_wildcard_configured_pattern(tags=tags, configured_tag=configured_tag) # True configured_tag = "User-Agent: python-requests/*" _tag_matches_wildcard_configured_pattern(tags=tags, configured_tag=configured_tag) # True configured_tag = "gm" _tag_matches_wildcard_configured_pattern(tags=tags, configured_tag=configured_tag) # False rN)PatternMatchRouterc3s|] }j|dVqdS))rVstringN)match)rrreZ regex_patternr"r#rXY sz;_tag_matches_wildcard_configured_pattern..)rZ.litellm.router_utils.pattern_match_deploymentsrZ_pattern_to_regexrZ)rrrZpattern_routerr"rr#(_tag_matches_wildcard_configured_pattern; s   rcCsddlm}tj}|dust|dkriSi}|D]%}|d|}||vr+d||<qd|vr:t||dr:d||<qd||<q|S) a Get custom labels from tags based on admin configuration. Supports both exact matches and wildcard patterns: - Exact match: "prod" matches "prod" exactly - Wildcard pattern: "User-Agent: curl/*" matches "User-Agent: curl/7.68.0" Reuses PatternMatchRouter for wildcard pattern matching. Returns dict of label_name: "true" if the tag matches the configured tag, "false" otherwise { "tag_User-Agent_curl": "true", "tag_User-Agent_python_requests": "false", "tag_Environment_prod": "true", "tag_Environment_dev": "false", "tag_Service_api_gateway_v2": "true", "tag_Service_web_app_v1": "false", } r)rNZtag_truer)rrfalse)rsrrrZcustom_prometheus_tagsrr)rrZconfigured_tagsrrZ label_namer"r"r#r\ s"   rrU)1r6rr@rrrrrrrrr r r r r rrrrrrrrZ'litellm.litellm_core_utils.core_helpersrrrrrrrrsrrrrZapscheduler.schedulers.asynciorr!r$r%rjrrxr#rr rrr"r"r"r#sz4  A 2  "!