o ưi22@sdZddlZddlZddlmZmZmZddlmZmZm Z ddl m Z ddl m Z ddlmZddlmZmZmZdd lmZdd lmZmZmZmZdd lmZmZmZd Zd Z d Z!d Z"GdddeZ#dS)a\ PagerDuty Alerting Integration Handles two types of alerts: - High LLM API Failure Rate. Configure X fails in Y seconds to trigger an alert. - High Number of Hanging LLM Requests. Configure X hangs in Y seconds to trigger an alert. Note: This is a Free feature on the regular litellm docker image. However, this is under the enterprise license N)datetime timedeltatimezone)ListOptionalUnion)verbose_logger) DualCache) SlackAlerting)AsyncHTTPHandlerget_async_httpx_clienthttpxSpecialProvider)UserAPIKeyAuth)AlertingConfigPagerDutyInternalEventPagerDutyPayloadPagerDutyRequestBody)CallTypesLiteralStandardLoggingPayload&StandardLoggingPayloadErrorInformation<iXcseZdZdZ d deeeefffdd ZddZ de d e d ed e d eee eeff d dZdeede fddZdeedededefddZ d!deeded eefddZdedefddZZS)"PagerDutyAlertingz Tracks failed requests and hanging requests separately. If threshold is crossed for either type, triggers a PagerDuty alert. N alerting_argsc spttd}|std||_|pi}t|dt|dt |dt |dt d|_ g|_ g|_dS)NZPAGERDUTY_API_KEYzPAGERDUTY_API_KEY is not setfailure_threshold failure_threshold_window_secondshanging_threshold_seconds hanging_threshold_window_seconds)rrrr)super__init__osgetenv ValueErrorapi_keyrget#PAGERDUTY_DEFAULT_FAILURE_THRESHOLD2PAGERDUTY_DEFAULT_FAILURE_THRESHOLD_WINDOW_SECONDS+PAGERDUTY_DEFAULT_HANGING_THRESHOLD_SECONDS2PAGERDUTY_DEFAULT_HANGING_THRESHOLD_WINDOW_SECONDSpagerduty_alerting_args_failure_events_hanging_events)selfrkwargsZ_api_key __class__r/home/app/Keep/.python/lib/python3.10/site-packages/litellm_enterprise/enterprise_callbacks/pagerduty/pagerduty.pyr3s0   zPagerDutyAlerting.__init__c sdttj}|d}|std|dpi}|dpi}|jtd!iddd|d|dd |d d |d d |d d |d d|dd|dd|dd|dd|dd|dd|dd|dd|dd|dd|dd|d|j dd} |j dd} |j |j| | ddId Hd S)"z Record a failure event. Only send an alert to PagerDuty if the configured *failure* threshold is exceeded in the specified window. Zstandard_logging_objectz9standard_logging_object is required for PagerDutyAlertingZerror_informationmetadatafailure_event_typeZfailed_response timestamp error_class error_codeerror_llm_provider llm_provideruser_api_key_hashuser_api_key_aliasuser_api_key_spenduser_api_key_max_budgetuser_api_key_budget_reset_atuser_api_key_org_iduser_api_key_team_iduser_api_key_project_iduser_api_key_user_iduser_api_key_team_aliasuser_api_key_end_user_iduser_api_key_user_emailuser_api_key_request_routeuser_api_key_auth_metadatarrrzHigh LLM API Failure Rateeventswindow_seconds threshold alert_prefixNr/) rnowrutcr#r!r)appendrr(!_send_alert_if_thresholds_crossed) r+r,Z response_obj start_timeend_timerLZstandard_logging_payload error_info_metarIrJr/r/r0async_log_failure_eventTs~                   z)PagerDutyAlerting.async_log_failure_eventuser_api_key_dictcachedata call_typereturncs$tdt|j||ddS)z Example of detecting hanging requests by waiting a given threshold. If the request didn't finish by then, we treat it as 'hanging'. z#Inside Proxy Logging Pre-call hook!) request_datarUN)rinfoasyncio create_taskhanging_response_handler)r+rUrVrWrXr/r/r0async_pre_call_hooks z%PagerDutyAlerting.async_pre_call_hookrZc std|jdtdt|jdtIdH|j|dIdHr'dS|j t didddt t jd d d d d d d |jd|jd|jd|jd|jrY|jnFdd|jd|jd|jd|jd|jd|jd|jd|jd|j|jdt}|jdt}|j|j||ddIdHdSd|jd|jd|jd|jd|jd|jd|jd|jd|j|jdt}|jdt}|j|j||ddIdHdS) z Checks if request completed by the time 'hanging_threshold_seconds' elapses. If not, we classify it as a hanging request. z/Inside Hanging Response Handler!..sleeping for rz secondsN)rZr2Zhanging_responser3r4ZHangingRequestr5r6r8r9r:r;r<r=r>r?r@rArBrCrDrErZhanging_threshold_failsz#High Number of Hanging LLM RequestsrGr/) rdebugr(r#r&r\sleepZ_request_is_completedr*rNrrrLrrMr"Z key_aliasZspendZ max_budgetZbudget_reset_at isoformatZorg_idZteam_idZ project_idZuser_idZ team_aliasZ end_user_idZ user_emailZ request_router1r'rO)r+rZrUrIrJr/r/r0r^s    z*PagerDutyAlerting.hanging_response_handlerrHrIrJrKc sttjt|dfdd|D}|||tdt |d|d|t ||kr\|j |dd}|d t |d |d }d |i}|j ||d IdH|dSdS)z 1. Prune old events 2. If threshold is reached, build alert, send to PagerDuty 3. Clear those events )secondscs"g|] }|dtjkr|qS)r3)r#rmin).0ecutoffr/r0 s"zGPagerDutyAlerting._send_alert_if_thresholds_crossed..zHave z events in the last z seconds. Threshold is ) max_errorsz: z in the last z seconds.Z recent_errors) alert_messagecustom_detailsN) rrLrrMrclearextendrr`len_build_error_summariessend_alert_to_pagerduty) r+rHrIrJrKZprunedZerror_summariesrlrmr/rgr0rOs&    z3PagerDutyAlerting._send_alert_if_thresholds_crossedrjrkcCs4|| d}g}|D] }|d||q |S)z Build short text summaries for the last `max_errors`. Example: "ValueError (code: 500, provider: openai)" Nr3)poprN)r+rHrkZrecentZ summariesZfer/r/r0rq s   z(PagerDutyAlerting._build_error_summariesrlrmc sz-td|ttjd}tt|ddd|d|jdd}|jd t |d d id Id HWSt yI}zt d|WYd }~d Sd }~ww)z Send [critical] Alert to PagerDuty https://developer.pagerduty.com/api-reference/YXBpOjI3NDgyNjU-pager-duty-v2-events-api zSending alert to PagerDuty: )r7criticalz LiteLLM AlertZLiteLLM)summaryZseveritysource componentrmtrigger)payloadZ routing_keyZ event_actionz'https://events.pagerduty.com/v2/enqueuez Content-Typezapplication/json)urljsonheadersNz"Error sending alert to PagerDuty: ) rr`r r ZLoggingCallbackrrr"postdict Exception exception)r+rlrmZ async_clientryrfr/r/r0rrs4 z)PagerDutyAlerting.send_alert_to_pagerduty)N)rj)__name__ __module__ __qualname____doc__rrrr~rrTrr rrstrr_r^rrintrOrqrr __classcell__r/r/r-r0r-sR!:  C ( r)$rr\rrrrtypingrrrZlitellm._loggingrZlitellm.cachingr Z1litellm.integrations.SlackAlerting.slack_alertingr Z&litellm.llms.custom_httpx.http_handlerr r r Zlitellm.proxy._typesrZ$litellm.types.integrations.pagerdutyrrrrZlitellm.types.utilsrrrr$r%r&r'rr/r/r/r0s"