o ưi_o@sJdZddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl Z ddl m Z ddlZddlmZddlmZddlmZdd lmZmZdd lmZmZmZmZmZdd lmZdd l m!Z!m"Z"m#Z#dd l$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3e/j4gZ5Gdddee3Z6dS)ay DataDog Integration - sends logs to /api/v2/log DD Reference API: https://docs.datadoghq.com/api/latest/logs `async_log_success_event` - used by litellm proxy to send logs to datadog `log_success_event` - sync version of logging to DataDog, only used on litellm Python SDK, if user opts in to using sync functions async_log_success_event: will store batch of DD_MAX_BATCH_SIZE in memory and flush to Datadog once it reaches DD_MAX_BATCH_SIZE or every 5 seconds async_service_failure_hook: Logs failures from Redis, Postgres (Adjacent systems), as 'WARNING' on DataDog For batching specific details see CustomBatchLogger class N)datetime)AnyDictListOptionalUnion)Response)verbose_logger)uuid)CustomBatchLogger)should_use_datadog_mockcreate_mock_datadog_client)get_datadog_hostnameget_datadog_serviceget_datadog_sourceget_datadog_tagsget_datadog_base_url_from_env)tracer)_get_httpx_clientget_async_httpx_clienthttpxSpecialProvider)IntegrationHealthCheckStatus) DD_ERRORSDD_MAX_BATCH_SIZE DataDogStatusDatadogInitParamsDatadogPayload"DatadogProxyFailureHookJsonMessage)ServiceLoggerPayload ServiceTypes)StandardLoggingPayload)AdditionalLoggingUtilscseZdZfddZdefddZdeddfdd Zd@d d Zd d Z ddZ dAde de de deedee f ddZddZddZddZdededefddZd ee e fd!e d"ejd#ejdef d$d%Zd&edefd'd(Z )    dBd*ed+eed,ee d"eeeefd#eeeefd-ee f d.d/Z )    dBd*ed+eed,ee d"eeeefd#eeeefd-ee f d0d1Z!d ee e fd!e d"ejd#ejdef d2d3Z"d4eddfd5d6Z#deeeeffd7d8Z$de%fd9d:Z&d;edd?Z'Z(S)C DataDogLoggerc szctdt|_|jrttd|}||ttj d|_ t d}|r3|j |dn|t}|rB|d|_t|_t|t|_tjd i||jtdWd Sty{}z tdt||d }~ww) a Initializes the datadog logger, checks if the correct env variables are set Required environment variables (Direct API): `DD_API_KEY` - your datadog api key `DD_SITE` - your datadog site, example = `"us5.datadoghq.com"` Optional environment variables (DataDog Agent): `LITELLM_DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"` `LITELLM_DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs) Note: We use LITELLM_DD_AGENT_HOST instead of DD_AGENT_HOST to avoid conflicts with ddtrace which automatically sets DD_AGENT_HOST for APM tracing. zDatadog: in init datadog loggerz6[DATADOG MOCK] Datadog logger initialized in mock mode)Z llm_providerZLITELLM_DD_AGENT_HOST) dd_agent_host /api/v2/logs) flush_lock batch_sizez.Datadog: Got exception on init Datadog client N)r debugr is_mock_moder _get_datadog_paramsupdaterrZLoggingCallback async_clientosgetenv_configure_dd_agent_configure_dd_direct_apir intake_urlr sync_clientasyncio create_taskZperiodic_flushLockr&super__init__r Exception exceptionstr)selfkwargsdict_datadog_paramsr$Z dd_base_urle __class__r([/home/app/Keep/.python/lib/python3.10/site-packages/litellm/integrations/datadog/datadog.pyr8KsF       zDataDogLogger.__init__returncCsLi}tjdur$ttjtrtj}|Sttjtr$tditj}|S)z Get the datadog_params from litellm.datadog_params These are params specific to initializing the DataDogLogger e.g. turn_off_message_logging Nr()litellmZdatadog_params isinstancer model_dumpr)r<r>r(r(rBr+s    z!DataDogLogger._get_datadog_paramsr$NcCsBtdd}d|d|d|_td|_td|jdS) z Configure DataDog Agent for log forwarding Args: dd_agent_host: Hostname or IP of DataDog agent ZLITELLM_DD_AGENT_PORTZ10518zhttp://:r% DD_API_KEYzDatadog: Using DD Agent at N)r.r/r2rHr r))r<r$Z dd_agent_portr(r(rBr0s  z!DataDogLogger._configure_dd_agentcCsTtdddur tdtdddurtdtd|_dtdd|_dS)z Configure direct DataDog API connection Raises: Exception: If required environment variables are not set rHNz)DD_API_KEY is not set, set 'DD_API_KEY=<>ZDD_SITEz+DD_SITE is not set in .env, set 'DD_SITE=<>zhttps://http-intake.logs.r%)r.r/r9rHr2)r<r(r(rBr1s  z&DataDogLogger._configure_dd_direct_apic spztd||||||IdHWdSty7}ztdt|dtWYd}~dSd}~ww)aF Async Log success events to Datadog - Creates a Datadog payload - Adds the Payload to the in memory logs queue - Payload is flushed every 10 seconds or when batch size is greater than 100 Raises: Raises a NON Blocking verbose_logger.exception if an error occurs 7Datadog: Logging - Enters logging function for model %sNDatadog Layer Error -  r r)_log_async_eventr9r:r; traceback format_excr<r= response_obj start_timeend_timer?r(r(rBasync_log_success_events z%DataDogLogger.async_log_success_eventc spztd||||||IdHWdSty7}ztdt|dtWYd}~dSd}~ww)NrIrJrKrLrPr(r(rBasync_log_failure_eventsz%DataDogLogger.async_log_failure_event request_dataoriginal_exceptionuser_api_key_dict traceback_strc szddlm}ddlm}|j||d}|dpd}d} |r,t|r,t |} i} zddl m } | j |d } t | trDt| n| } Wn=tyt|d r[t|d d| d <t|d rht|d d| d <t|d rut|d d| d <t|d rt|d d| d <Ynw|dpt||dp|jj| |dpd| d} tttt|| ttjd}|j|d|j|t|j|jkr| IdHWdSWdSty}zt!"dt|dt#$WYd}~dSd}~ww)a Log proxy-level failures (e.g. 401 auth, DB connection errors) to Datadog. Ensures failures that occur before or outside the LLM completion flow (e.g. ConnectError during auth when DB is down) are visible in Datadog alongside Prometheus. r)StandardLoggingPayloadSetup safe_dumps)rWrYZ error_codeN)LiteLLMProxyRequestSetup)rXZ request_routeZteam_idZuser_idZ end_user_id error_message error_classrN)r:r` status_coderNrXZddsourceZddtagshostnamemessageservicestatus dd_payloadz(Datadog: async_post_call_failure_hook - rK)%*litellm.litellm_core_utils.litellm_loggingrZ*litellm.litellm_core_utils.safe_json_dumpsr\Zget_error_informationgetr;stripisdigitintZ$litellm.proxy.litellm_pre_call_utilsr^Z'get_sanitized_user_information_from_keyrEdictr9hasattrgetattrrA__name__rrrrrrERROR_add_trace_context_to_payload log_queueappendlenr'async_send_batchr r:rNrO)r<rVrWrXrYrZr\Zerror_information_coderaZ user_contextr^_metaZmessage_payloadrhr?r(r(rBasync_post_call_failure_hooks              z*DataDogLogger.async_post_call_failure_hookc szi|js tdWdStdt|j|j|jr td||jIdH}|jdkr8tt j j WdS| |jdkrMt d|jd|j|jr_td t|jd WdStd |j|jWdSt y}ztd t|d tWYd}~dSd}~ww)z Sends the in memory logs queue to datadog api Logs sent to /api/v2/logs DD Ref: https://docs.datadoghq.com/api/latest/logs/ Raises: Raises a NON Blocking verbose_logger.exception if an error occurs z!Datadog: log_queue does not existNz(Datadog - about to flush %s events on %sz@[DATADOG MOCK] Mock mode enabled - API calls will be interceptedi'Response from datadog API status_code: , text: z[DATADOG MOCK] Batch of z events successfully mockeds8