o ưi @sddlZddlmZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZm Z erdd l!m"Z#dd l$m%Z&ddl$m'Z(ddl$m)Z*ddl+m,Z-ddl.m/Z0e e(efZ'e e*efZ)e e&efZ%e e#efZ"e e0efZ/e e-efZ,n eZ'eZ)eZ"eZ/eZ,eZ%e1ddZ2e1ddZ3e1ddZ4dZ5dZ6dZ7eGdddZ8GdddeZ9dS)N) dataclass)datetime) TYPE_CHECKINGAnyDictListOptionalUnioncast)verbose_logger)OpenInferenceSpanKindValuesSpanAttributes) CustomLogger) safe_dumps)get_secret_bool)ServiceLoggerPayload)ChatCompletionMessageToolCall CostBreakdownFunctionLLMResponseTypesStandardCallbackDynamicParamsStandardLoggingPayload) SpanExporter)Context)Span)Tracer) ManagementEndpointLoggingPayload)UserAPIKeyAuthZOTEL_TRACER_NAMElitellmLITELLM_METER_NAMELITELLM_LOGGER_NAMEzReceived Proxy Server RequestZraw_gen_ai_requestZlitellm_requestc@seZdZUdZeeefed<dZe eed<dZ e eed<dZ e ed<dZ e ed<dZe eed <dZe eed <dZe eed <dd dZeddZdS)OpenTelemetryConfigconsoleexporterNendpointheadersFenable_metrics enable_events service_namedeployment_environmentmodel_idreturncCsh|jrt|jtr|jdkrd|_|jstdd|_|js%tdd|_|js2td|j|_dSdS)Nr" otlp_httpOTEL_SERVICE_NAMErOTEL_ENVIRONMENT_NAME production OTEL_MODEL_ID) r$ isinstancer#strr(osgetenvr)r*)selfr6Y/home/app/Keep/.python/lib/python3.10/site-packages/litellm/integrations/opentelemetry.py __post_init__Hs  z!OpenTelemetryConfig.__post_init__c Csddlm}tdtdd}tdtd}tdtd }td d d k}td d d k}tdd}tdd}td|} |dkrR||dS||||||||| dS)z OTEL_HEADERS=x-honeycomb-team=B85YgLm9**** OTEL_EXPORTER="otlp_http" OTEL_ENDPOINT="https://api.honeycomb.io/v1/traces" OTEL_HEADERS gets sent as headers = {"x-honeycomb-team": "B85YgLm96******"} r)InMemorySpanExporterZOTEL_EXPORTER_OTLP_PROTOCOL OTEL_EXPORTERr"ZOTEL_EXPORTER_OTLP_ENDPOINT OTEL_ENDPOINTZOTEL_EXPORTER_OTLP_HEADERS OTEL_HEADERSZ'LITELLM_OTEL_INTEGRATION_ENABLE_METRICSfalsetrueZ&LITELLM_OTEL_INTEGRATION_ENABLE_EVENTSr-rr.r/r0Z in_memory)r#)r#r$r%r&r'r(r)r*)Z6opentelemetry.sdk.trace.export.in_memory_span_exporterr9r3r4lower) clsr9r#r$r%r&r'r(r)r*r6r6r7from_env]s>       zOpenTelemetryConfig.from_env)r+N)__name__ __module__ __qualname__r#r r2r__annotations__r$rr%r&boolr'r(r)r*r8 classmethodrAr6r6r6r7r!=s    r!cseZdZ     ddeedeedeedeedeef fdd Zedefd d Z d d Z ddede fddZ ddZ ddZddZddZddZddZddZ    dd ed!eed"eeeefd#eeeefd$eef d%d&Z '    dd ed(eed!eed"eeeefd#eeeefd$eef d)d*Z dd+ed,ed-ed.eefd/d0Zd1ed-ed2efd3d4Z d5ed6e!fd7d8Z"d6eefd9d:Z#d;efde%d6eefd?d@Z&dAdBZ'dCdDZ(dEdFZ)dGdHZ*edIeeeeefd6eefdJdKZ+d5edLefdMdNZ,d5edOeed#edPedLef dQdRZ-d5ed#eeefdLefdSdTZ.dUefdVdWZ/edUeedXeedYeed6eefdZd[Z0d5eed\ee1fd]d^Z2d_d`Z3dUed5efdadbZ4dUefdcddZ5d6eee e6effdedfZ7edge8e9d6e:eeffdhdiZ;dUedOeefdjdkZdree8eefd6e8efdsdtZ?due8ed6e8efdvdwZ@dUefdxdyZAdzd{ZBd|d}ZCd~dZDddeefddZEdd;eefddZFddZGddZHdeeded6eefddZIedeeeefd6e:eeffddZJ ddeKd!eefddZL ddeKd!eefddZMd"eded6eefddZNZOS) OpenTelemetryNconfig callback_nametracer_providerlogger_providermeter_providerc  s|durt}||_||_|jj|_|jj|_|jj|_ i|_ | |t t dd}|dkrNddl}|j|jd|t|d} | |jtjdi||||||dS)NZ DEBUG_OTELFalser>r)levelzopentelemetry.sdk.trace.exportr6)r!rArIrJr#r:r$r;r%r<_tracer_provider_cache _init_tracingr2r3r4r?logging basicConfigDEBUG getLoggerrBsetLevelsuper__init__ _init_metrics _init_logs"_init_otel_logger_on_litellm_proxy) r5rIrJrKrLrMkwargsZ _debug_otelrRZotel_exporter_logger __class__r6r7rXs(           zOpenTelemetry.__init__cCsJddlm}m}|j|j|jp|jd}||}|}|}||S)z>Create an OpenTelemetry Resource using config-driven defaults.r)OTELResourceDetectorResource)z service.namezdeployment.environmentr*) Zopentelemetry.sdk.resourcesr_r`r(r)r*createdetectmerge)rIr_r`Zbase_attributesZ base_resourceZotel_resource_detectorZ env_resourcer6r6r7_get_litellm_resources   z#OpenTelemetry._get_litellm_resourcecCsjzddlm}WntytdYdSwdtjvr-tddtjDr-tj|t |d|dS) z Initializes OpenTelemetry for litellm proxy server - Adds Otel as a service callback - Sets `proxy_server.open_telemetry_logger` to self r) proxy_serverzEProxy Server is not installed. Skipping OpenTelemetry initialization.NZotelcss|] }t|t VqdSN)r1rH).0cbr6r6r7 s  zCOpenTelemetry._init_otel_logger_on_litellm_proxy..Zopen_telemetry_logger) Z litellm.proxyre ImportErrorr warningrZservice_callbackallappendsetattr)r5rer6r6r7r[s  z0OpenTelemetry._init_otel_logger_on_litellm_proxyF provider_nameskip_set_globalc Cs|durtdt|j|Sz2|}t||r'td|t|j|}W|Std||}|s9||W|Std|W|Stym} z td|t| |}|sb||WYd} ~ |SWYd} ~ |Sd} ~ ww)a Generic helper to get or create an OpenTelemetry provider (Tracer, Meter, or Logger). Args: provider: The provider instance passed to the init function (can be None) provider_name: Name for logging (e.g., "TracerProvider") get_existing_provider_fn: Function to get the existing global provider sdk_provider_class: The SDK provider class to check for (e.g., TracerProvider from SDK) create_new_provider_fn: Function to create a new provider instance set_provider_fn: Function to set the provider globally skip_set_global: If True, don't set the provider globally (for dynamic-only providers) Returns: The provider to use (either existing, new, or explicitly provided) Nz0OpenTelemetry: Using provided TracerProvider: %sz$OpenTelemetry: Using existing %s: %szOpenTelemetry: Creating new %sz^OpenTelemetry: Created %s but NOT setting it globally (will use dynamic providers per-request)zCOpenTelemetry: Exception checking existing %s, creating new one: %s)r debugtyperBr1info Exceptionr2) r5providerroget_existing_provider_fnsdk_provider_classcreate_new_provider_fnset_provider_fnrpZexisting_providerer6r6r7_get_or_create_providersP    z%OpenTelemetry._get_or_create_providerc sxddlm}ddlmddlm}fdd}tdo"jdk}j|d |j ||j |d }| t _ |_dS) NrtraceTracerProvider)SpanKindcs$jd}||SNresource)rdrIadd_span_processor_get_span_processor)rurr5r6r7create_tracer_provider(sz;OpenTelemetry._init_tracing..create_tracer_providerrJ langfuse_otelr)rurorvrwrxryrp) opentelemetryr}opentelemetry.sdk.traceropentelemetry.tracerhasattrrJr{Zget_tracer_providerZset_tracer_provider get_tracerLITELLM_TRACER_NAMEtracer span_kind)r5rKr}rrZ skip_globalr6rr7rQ#s"    zOpenTelemetry._init_tracingcsjjsd_d_d_d_d_d_dSddlm }ddl m fdd}j |d|j ||jd}|t}|jdd d d _|jd d dd _|jdddd _|jddd d _|jddd d _|jddd d _dS)Nr)metrics) MeterProvidercs}|gjdS)N)Zmetric_readersr)_get_metric_readerrdrI)Z metric_readerrr5r6r7create_meter_providerNs  z:OpenTelemetry._init_metrics..create_meter_providerrrurorvrwrxryz gen_ai.client.operation.durationzGenAI operation durations)name descriptionunitzgen_ai.client.token.usagezGenAI token usagez{token}zgen_ai.client.token.costzGenAI request costUSDz*gen_ai.client.response.time_to_first_tokenz*Time to first token for streaming requestsz,gen_ai.client.response.time_per_output_tokenzCAverage time per output token (generation time / completion tokens)zgen_ai.client.response.durationz9Total LLM API generation time (excludes LiteLLM overhead))rIr&_operation_duration_histogram_token_usage_histogram_cost_histogram_time_to_first_token_histogram _time_per_output_token_histogram_response_duration_histogramrropentelemetry.sdk.metricsrr{Zget_meter_providerZset_meter_providerZ get_meterrBZcreate_histogram)r5rMrrZmeterr6rr7rYAsd    zOpenTelemetry._init_metricscs^jjsdSddlm}m}ddlmddlmfdd}j |d|||ddS) Nr)get_logger_providerset_logger_provider)LoggerProvider)BatchLogRecordProcessorcs,jd}}|||Sr)rdrI_get_log_exporterZadd_log_record_processor)ruZ log_exporterrZOTLoggerProviderr5r6r7create_logger_providers z8OpenTelemetry._init_logs..create_logger_providerrr) rIr'opentelemetry._logsrropentelemetry.sdk._logsropentelemetry.sdk._logs.exportrr{)r5rLrrrr6rr7rZs    zOpenTelemetry._init_logscC|||||dSrf_handle_successr5r\ response_obj start_timeend_timer6r6r7log_success_eventzOpenTelemetry.log_success_eventcCrrf_handle_failurerr6r6r7log_failure_eventrzOpenTelemetry.log_failure_eventc|||||dSrfrrr6r6r7async_log_success_eventz%OpenTelemetry.async_log_success_eventcrrfrrr6r6r7async_log_failure_eventrz%OpenTelemetry.async_log_failure_eventpayloadparent_otel_spanrrevent_metadatac s:ddlm}ddlm}m}d} d} t|trt|d} n||} t|tr0t|d} n||} |dur|j } |j j | | || d} |j | d|jd|j | d|j jd|r|D])\} }|durkd }t|trzt|}Wn tyd }Ynw|j | | |dqa| ||j| j| d dSdS) Nrr|Status StatusCodeeArcontextr call_typespankeyvalueserviceNonez0litellm logging error - could_not_json_serializer)rr}rrrr1floatint_to_nsrr start_spanset_span_in_contextsafe_set_attributerritemsdictr2rt set_statusOKend)r5rrrrrr}rr_start_time_ns _end_time_ns _span_nameservice_logging_spanrrr6r6r7async_service_success_hooks\        z(OpenTelemetry.async_service_success_hookerrorc sJddlm}ddlm}m} d} d} t|tr!tt|d} n||} t|tr4tt|d} n||} |dur|j } |j j | | || d} |j | d|jd|j | d|j jd|ri|j | d |d|r|D]#\}}t|trzt|}Wn tyd }Ynw|j | ||dqo| || j| j| d dSdS) Nrr|rrrrrrrz/litllm logging error - could_not_json_serializer)rr}rrrr1rrrrrrrrrrrrr2rtrERRORr)r5rrrrrrr}rrrrrrrrr6r6r7async_service_failure_hooksd        z(OpenTelemetry.async_service_failure_hook request_dataoriginal_exceptionuser_api_key_dict traceback_strc sddlm}ddlm}m}|j}|durU|||jd} |jj | | |d} |j | dt |d| ||j| j |td|j |tddSdS) Nrr|rzFailed Proxy Server Request)rr exceptionrr)rr}rrrrrrrrrrr2rrrnow) r5rrrrr}rrrrZexception_logging_spanr6r6r7async_post_call_failure_hook"s( z*OpenTelemetry.async_post_call_failure_hookdataresponsec sZddlm}|d}|dur+t||r+|j}|j}|j||d\}} |j||d|S)Nr)Logginglitellm_logging_obj) default_spanr\r)Z*litellm.litellm_core_utils.litellm_loggingrgetr1Zmodel_call_detailsr_get_span_context_create_guardrail_span) r5rrrZLiteLLMLoggingrr\ parent_spanctx_r6r6r7async_post_call_success_hookAs   z*OpenTelemetry.async_post_call_success_hookr\r+cCs||}|dur||}td||St|drC|jdkrC|jr)||jni}|r9||}td|S|j}td|S|j}td|S)a Get the tracer to use for this request If dynamic headers are present, a temporary tracer is created with the dynamic headers. Otherwise, the default tracer is used. Returns: Tracer: The tracer to use for this request Nz2[OTEL DEBUG] Using DYNAMIC tracer with headers: %srJrzM[OTEL DEBUG] Using env var credentials for langfuse_otel (master key request)z7[OTEL DEBUG] No credentials available for langfuse_otelz5[OTEL DEBUG] Using GLOBAL tracer (no dynamic headers)) %_get_dynamic_otel_headers_from_kwargs _get_tracer_with_dynamic_headersr rqrrJr<_get_headers_dictionaryr)r5r\dynamic_headersZ tracer_to_useZenv_var_headersr6r6r7get_tracer_to_use_for_request[s<   z+OpenTelemetry.get_tracer_to_use_for_requestcCs*|d}|s dS|j|d}|r|SdS)z1Extract dynamic headers from kwargs if available. standard_callback_dynamic_paramsN)r)rconstruct_dynamic_otel_headers)r5r\rrr6r6r7rs z3OpenTelemetry._get_dynamic_otel_headers_from_kwargsrcCsnddlm}tt|}||jvr|j|tS|||j d}| |j |d||j|<|tS)zECreate a temporary tracer with dynamic headers for this request only.rr~r)r) rrr2sortedrrPrrrdrIrr)r5rr cache_keyZ temp_providerr6r6r7rs      z.OpenTelemetry._get_tracer_with_dynamic_headersrcCsdS)a Construct dynamic headers from standard callback dynamic params Note: You just need to override this method in Arize, Langfuse Otel if you want to allow team/key based logging. Returns: dict: A dictionary of dynamic headers Nr6)r5rr6r6r7rs z,OpenTelemetry.construct_dynamic_otel_headerscCs`td||j||\}}t|do|jdk}|rd}d}|dup&td}|r<||||||} |||||| n'ddl m } m } d} t|drZ| | | j |||||||||||j| ||d} |j|| d ||||||jjr| dur| n|} | dur|||| |durt|d r|jtkr|j||d dSdSdSdS) NzAOpenTelemetry Logger: Logging kwargs: %s, OTEL config settings=%srJrUSE_OTEL_LITELLM_REQUEST_SPANrrrrr fallback_ctxrrr)r rqrIrrrJr_start_primary_span_maybe_log_raw_requestrrrrrset_attributes_resolve_guardrail_contextr_record_metricsr'_emit_semantic_logsrLITELLM_PROXY_REQUEST_SPAN_NAMErr)r5r\rrrrris_langfuse_otelshould_create_primary_spanrrr guardrail_ctxZlog_spanr6r6r7rsZ       zOpenTelemetry._handle_successc Csjddlm}m}||}|j|||||d} | ||j| | ||| j ||d| S)Nrrrrrr) rrrrr_get_span_namerrrrr) r5r\rrrrrr otel_tracerrr6r6r7rs z!OpenTelemetry._start_primary_spancCsddlm}ddlm}m}tjs|jsdS|di} | dp"i} | d} | r,| nt } | |} | j | | || |d}|||j|||||j| |ddS) Nrr|rlitellm_paramsmetadatageneration_namer r)rr}rrrrturn_off_message_loggingmessage_loggingrRAW_REQUEST_SPAN_NAMErrrrrrset_raw_request_attributesr)r5r\rrrrr}rrrrrZ raw_span_namerZraw_spanr6r6r7r(s"      z$OpenTelemetry._maybe_log_raw_requestcCs||}|dp i}|dd}d||ddd}|d} t| d dp/| p+id i} d D]} | | durFt| | |d | <q2t| d dpT| pPid i} | r]t| |d <|jr|jj||d |r|d} r|jri|ddi}i|ddi}|jj| dd|d |jj| dd|d |d}|jr|r|jj||d | ||| |||||| |||dS)Nrcustom_llm_providerUnknownchatmodelr)zgen_ai.operation.name gen_ai.systemzgen_ai.request.modelzgen_ai.frameworkstandard_logging_objectr)Zuser_api_key_hashZuser_api_key_aliasZuser_api_key_team_idZuser_api_key_org_idZuser_api_key_user_idZuser_api_key_team_aliasZuser_api_key_user_emailZspend_logs_metadataZrequester_ip_addressZrequester_metadataZuser_api_key_end_user_idZprompt_management_metadataZapplied_guardrailsZmcp_tool_call_metadataZvector_store_request_metadataz metadata. hidden_params attributesusagezgen_ai.token.typeinputoutput prompt_tokensrcompletion_tokensZ response_cost) total_secondsrgetattrr2rrrecordrr"_record_time_to_first_token_metric$_record_time_per_output_token_metric _record_response_duration_metric)r5r\rrr duration_sparamsru common_attrsZstd_logmdrrrZin_attrsZ out_attrsZcostr6r6r7rBsZ           zOpenTelemetry._record_metricsvalc Cs|durdSt|tr|St|ttfrt|Sz t|dWStyAz t|dWYSty@YYdSww)z+Convert datetime/float/string to timestamp.Nz%Y-%m-%d %H:%M:%S.%fz%Y-%m-%d %H:%M:%S)r1r timestamprrstrptime ValueError)r.r6r6r7 _to_timestamps   zOpenTelemetry._to_timestampr,c Cs|di}|dd}|jr|sdS|dd}|dd}|durI|durK||}||}|dus9|dur;dS||} |jj| |ddSdSdS)z@Record Time to First Token (TTFT) metric for streaming requests.optional_paramsstreamFNapi_call_start_timecompletion_start_timer)rrr2r&) r5r\r,r3Z is_streamingr5r6api_call_start_tscompletion_start_tsZtime_to_first_token_secondsr6r6r7r's        z0OpenTelemetry._record_time_to_first_token_metricrr*cCs|jsdSd}|r|d}r|d}|dus|dkrdS|dd}|dd} ||} | durH|} | dkrF| |} |jj| |ddS|dur]||} | durX|} n| | } n| durr|| }|durm|} n| |} n|} | dkr| |} |jj| |ddSdS)a2Record Time Per Output Token (TPOT) metric. Calculated as: generation_time / completion_tokens - For streaming: uses end_time - completion_start_time (time to generate all tokens after first) - For non-streaming: uses end_time - api_call_start_time (total generation time) Nrr#rr6r5r)rrr2r&)r5r\rrr*r,r#rr6r5 end_time_tsZgeneration_time_secondsZtime_per_output_token_secondsr8r7r6r6r7r(sH         z2OpenTelemetry._record_time_per_output_token_metricc Cs|jsdS|dd}|durdS|dp|}|dur t}||}||}|dus2|dur4dS||}|dkrF|jj||ddSdS)aRecord Total Generation Time (response duration) metric. Measures pure LLM API generation time: end_time - api_call_start_time This excludes LiteLLM overhead and measures only the LLM provider's response time. Works for both streaming and non-streaming requests. Mirrors Prometheus's litellm_llm_api_latency_metric. Uses kwargs.get("end_time") with fallback to parameter for consistency with Prometheus. Nr5rrr)rrrrr2r&) r5r\rr,r5Z _end_timer7r9Zresponse_duration_secondsr6r6r7r)s$    z.OpenTelemetry._record_response_duration_metricrc Cs|jjsdSddlm}m}zddlm}Wnty%ddlm}Ynw|t }| }| dp4i dd} | dgD]F} | dd } d | d } | d kr[| d r[| d | d <|j ri| dri| d| d<|| t|j|j|j|jd| | d} || q?t| dgD][\}}d| || dd} | di}|j r| dr|d| d<|| dd| ddid}|j r| dr|d|dd<|| t|j|j|j|jd|| d} || qdS)Nr)SeverityNumber get_logger) LogRecordrrrmessagesroleuserzgen_ai.content.prompt) event_namertoolidcontentz gen_ai.promptINFO)r/trace_idspan_id trace_flagsZseverity_numberZ severity_textbodyrchoiceszgen_ai.content.completion finish_reason)r@rindexrJmessagezmessage.contentZ assistant)rKrJrL)rIr'rr:r;rr<rjZ!opentelemetry.sdk._logs._internalr get_span_contextrrrrrrErFrGrDcopyemit enumerate)r5r\rrr:r;Z SdkLogRecordZ otel_loggerZ parent_ctxrumsgr>attrsZ log_recordidxchoiceZbody_msgrHr6r6r7r'sx           z!OpenTelemetry._emit_semantic_logsrrcCs4ddlm}|dur||S|dur||S|S)u1 Return a valid OTEL context for guardrail child spans so they are never orphaned (Issue #5). Priority: 1. The litellm_request span that was just created 2. The parent proxy-request span 3. The original fallback context (may be None — last resort) rr|N)rr}r)rrr_tracer6r6r7r}s   z(OpenTelemetry._resolve_guardrail_contextrcCsH|pi}|d}|durdS|d}|sdSdd|D}|s#dS||}|D]w}|d}|d} t} |durCt|} t} | durPt| } |jd|| |d } |j| tj t j j d |j| d |d d |j| d |d d |d } | dur| d t| |j| d|dd | j|| dq*dS)zr Creates a span for Guardrail, if any guardrail information is present in standard_logging_object rNguardrail_informationcSsg|] }t|tr|qSr6)r1r)rg informationr6r6r7 s z8OpenTelemetry._create_guardrail_span..rrZ guardrailr rZguardrail_nameZguardrail_modemasked_entity_countZguardrail_responser)rrrr fromtimestamprrrr ZOPENINFERENCE_SPAN_KINDr Z GUARDRAILr set_attributerr)r5r\rstandard_logging_payloadZguardrail_information_dataZguardrail_information_listrrVZstart_time_floatZend_time_floatZstart_time_datetimeZend_time_datetimeZguardrail_spanrYr6r6r7rst      z$OpenTelemetry._create_guardrail_spancCs\ddlm}m}td||j||\}}t|do |jdk} | r'd}d}|dup.t d} d} | rf| |} | j | || ||d} | ||j|| |||j| |d| j| |d n|r|||j|||||j||d|j| ||d } |j|| d |durt|d r|jtkr|j| |d dSdSdSdS) NrrzPOpenTelemetry Logger: Failure HandlerLogging kwargs: %s, OTEL config settings=%srJrrr )rr\rrrr)rrrr rqrIrrrJrrrr rrrr_record_exception_on_spanr is_recordingrrrr)r5r\rrrrr_parent_contextrr r rrr r6r6r7rsT   zOpenTelemetry._handle_failurec Cs\zddlm}|d}|dur|||d}|dur!WdS|d}|dur=|d}|r:|j||j|dWdS|d rM|j||j|d d|d r]|j||j|d d|d rm|j||j|d d|d r}|j||j|d d|d r|j||j |d dWdSWdSt y}zt dt |WYd}~dSd}~ww)aQ Record exception information on the span using OTEL standard methods. This extracts error information from StandardLoggingPayload and: 1. Uses span.record_exception() for the actual exception object (OTEL standard) 2. Sets structured error attributes from StandardLoggingPayloadErrorInformation r)ErrorAttributesrNrerror_information error_strrZ error_codeZ error_class error_messageZ llm_provider tracebackz4OpenTelemetry: Error recording exception on span: %s)*litellm.integrations._types.open_inferencer`rZrecord_exceptionrZ ERROR_MESSAGEZ ERROR_CODEZ ERROR_TYPEZERROR_LLM_PROVIDERZERROR_STACK_TRACErtr rr2) r5rr\r`rr\rarbrzr6r6r7r]%sr           z'OpenTelemetry._record_exception_on_spanc Csddl}ddlm}|sdSzJt|D]B\}}|d}|sq|jjd|}|j||d|dd|j||d|d d|j||d ||d dqWdSt ys} zt d t | WYd} ~ dSd} ~ ww) Nrr function.z.namerrz .descriptionrz .parameters parametersz1OpenTelemetry: Error setting tools attributes: %s) jsonlitellm.proxy._typesr rPrZLLM_REQUEST_FUNCTIONSrrdumpsrtr rr2) r5rtoolsrjr irArgprefixrzr6r6r7set_tools_attributesxsB  z"OpenTelemetry.set_tools_attributescCB|durdSt|ttttfr|Szt|WSty YdSwz Casts the value to a primitive OTEL type if it is not already a primitive type. OTEL supports - str, bool, int, float If it's not a primitive type, then it's converted to a string Nrr1r2rFrrrtr5rr6r6r7cast_as_primitive_value_type  z*OpenTelemetry.cast_as_primitive_value_type tool_callsc Cstddlm}i}t|D]+\}}|d}|sq tj}|D]}||}|r6|||jjd|d|<qq |S)Nrrfrgrhz.function_call.) rkr rPrrrEkeysZLLM_COMPLETIONSr) rwr kv_pairsrSZ tool_callZ _functionrxr_valuer6r6r7_tool_calls_kv_pairs"    z!OpenTelemetry._tool_calls_kv_pairc Csz|jdkrddlm}||||WdS|jdkr-ddlm}||||WdS|jdkrAddlm}||||WdSddl m }| d i}| d ipTi} | d } | durbt d | d } | D]\} } |j|d| | dqjt| ddp| pi di}|r|j|dt|d| d}|r| D]\} } | dur|j|d| | dq| dr|j||jj| dd|j||jj| dd|j||jj| ddd| dr|j||jj| dd| dr|j||jj| dd| dr|j||jj| dd|j||jjt| ddd| dr:|j||jj| dd|rB| dndpI| d}|rU|j|d|d|rk| drk|j||jj| dd|or| d}|r|j||jj| d d|j||jj| d!d|j||jj| d"dt j!d#urWdS|j"d#urWdS| d$r|d$}|#||| d%r|$| d%}|j||j%jt|d| d&r|$| d&}|j||j&jt|d|j||j'j| dd'krd(n| dp d(d| d)r |j||j(j| d)d|dur| d*r|)| d*}|j||j*jt|dg}t+| d*D]\}}| d+rZ|,| d+qH|rk|j||j-jt|dt+| d*D]5\}}| d+r| d,}| d-}|rt./|}| D] \} } |j|| | dqqrWdSWdSWdSt0y}z|j1|jpd.d/t23d0t|WYd}~dSd}~ww)1NZ langtracer)LangtraceAttributesr)LangfuseOtelLoggerZ weave_otel)set_weave_otel_attributesrfr3rrz+standard_logging_object not found in kwargsrz metadata.{}rrcost_breakdownz gen_ai.cost.rrrrZ max_tokensZ temperatureZtop_pr4Fr?rBzgen_ai.response.idrZ total_tokensr#r"Trmr=Zsystem_instructions completionrZ request_idrIrJrLrwr)rJz0OpenTelemetry logging error in set_attributes %s)4rJZlitellm.integrations.langtracer|Zset_langtrace_attributesZ+litellm.integrations.langfuse.langfuse_otelr}Zset_langfuse_otel_attributesZ%litellm.integrations.weave.weave_otelr~rkr rr1rrformatr%rZLLM_REQUEST_MODELrZLLM_REQUEST_TYPEZ LLM_SYSTEMZLLM_REQUEST_MAX_TOKENSZLLM_REQUEST_TEMPERATUREZLLM_REQUEST_TOP_PZLLM_IS_STREAMINGr2ZLLM_USERZLLM_RESPONSE_MODELZGEN_AI_USAGE_TOTAL_TOKENSZGEN_AI_USAGE_OUTPUT_TOKENSZGEN_AI_USAGE_INPUT_TOKENSrrrrp0_transform_messages_to_otel_semantic_conventionsZGEN_AI_INPUT_MESSAGESZGEN_AI_SYSTEM_INSTRUCTIONSZGEN_AI_OPERATION_NAMEZGEN_AI_REQUEST_ID/_transform_choices_to_otel_semantic_conventionsZGEN_AI_OUTPUT_MESSAGESrPrmZGEN_AI_RESPONSE_FINISH_REASONSrHr{rtZhandle_callback_failurer r)r5rr\rr|r}r~r r3rr\rrrrrZ response_idrrmZtransformed_messagesZtransformed_system_instructionsZtransformed_choicesZfinish_reasonsrSrTrLrwryrzr6r6r7rs                                  zOpenTelemetry.set_attributescCrqrrrsrtr6r6r7_cast_as_primitive_value_typervz+OpenTelemetry._cast_as_primitive_value_typerrcCs||}|||dS)z_ Safely sets an attribute on the span, ensuring the value is a primitive type. N)rr[)r5rrrZprimitive_valuer6r6r7rs z OpenTelemetry.safe_set_attributer=c Cst|trdd|dgdgSg}|D]e}|dd}|dd}g}t|tr1|d|dnt|trP|D]}t|trE||q8|dt|dq8||d}d |vr_|d |d <d |vri|d |d <d |vrs|d |d <||q|S) z Transforms LiteLLM/OpenAI style messages into OTEL GenAI 1.38 compliant format. OTEL expects a 'parts' array instead of a single 'content' string. systemtext)rrrC)r>partsr>r?rCrrBrwZ tool_call_id)r1r2rrmlistr) r5r= transformedrQr>rCrparttransformed_msgr6r6r7rs4             z>OpenTelemetry._transform_messages_to_otel_semantic_conventionsrIcCsPg}|D]!}|dp i}|d}||gd}|r ||d<||q|S)z_ Transforms choices into OTEL GenAI 1.38 compliant format for output.messages. rLrJr)rrrm)r5rIrrTrLrJrr6r6r7rs  z=OpenTelemetry._transform_choices_to_otel_semantic_conventionsc CsXz|dipi}|dd}|d}|dipi}|d}|r?t|tr?|D]\} } |j|d|d| | d q,|rt|trd dl} z | |}|D]\} } |j|d|d| | d qTWWdS| jyt d ||j|d|d |d YWdSwWdSWdSt y} zt d t| WYd} ~ dSd} ~ ww)Nrrroriginal_responseZadditional_argscomplete_input_dictzllm.rhrrzglitellm.integrations.opentelemetry.py::set_raw_request_attributes() - raw_response not json string - {}z.stringified_raw_responsezs$z5OpenTelemetry._get_span_processor..z8[OTEL DEBUG] Creating span processor with GLOBAL headersexportzDOpenTelemetry: intiializing SpanExporter. Value of OTEL_EXPORTER: %sr"zHOpenTelemetry: intiializing console exporter. Value of OTEL_EXPORTER: %sr, http/protobuf http/json)OTLPSpanExporterzmOpenTelemetry OTLP HTTP exporter is not available. Install `opentelemetry-exporter-otlp` to enable OTLP HTTP.zEOpenTelemetry: intiializing http exporter. Value of OTEL_EXPORTER: %stracesr$r% otlp_grpcgrpcz{OpenTelemetry OTLP gRPC exporter is not available. Install `opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`).zEOpenTelemetry: intiializing grpc exporter. Value of OTEL_EXPORTER: %s)opentelemetry.sdk.trace.exportrrrrr rqr:r;r<rHrrrr Z5opentelemetry.exporter.otlp.proto.http.trace_exporterrrj_normalize_otel_endpointZ5opentelemetry.exporter.otlp.proto.grpc.trace_exporter) r5rrrrr_split_otel_headersZOTLPSpanExporterHTTPexcnormalized_endpointZOTLPSpanExporterGRPCr6r6r7rs      z!OpenTelemetry._get_span_processorc Csftd|j|j|jt|j}||jd}td|j|t|jdr0td|j|jSt d}|jdks>|dkrNdd l m }td |j|S|jd ks]|jd ks]|jd krqddl m}td|j||||dS|jdks{|jdkrzddlm}Wnty}ztd|d}~wwtd|j||||dStd|jdd l m }|S)zN Get the appropriate log exporter based on the configuration. ztOpenTelemetry Logger, initializing log exporter self.OTEL_EXPORTER: %s self.OTEL_ENDPOINT: %s self.OTEL_HEADERS: %slogsz4OpenTelemetry: Log endpoint normalized from %s to %srzDOpenTelemetry: Using custom log exporter. Value of OTEL_EXPORTER: %sZOTEL_LOGS_EXPORTERr"r)ConsoleLogExporterzEOpenTelemetry: Using console log exporter. Value of OTEL_EXPORTER: %sr,rr)OTLPLogExporterzPOpenTelemetry: Using HTTP log exporter. Value of OTEL_EXPORTER: %s, endpoint: %srrrzOpenTelemetry OTLP gRPC log exporter is not available. Install `opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`).NzPOpenTelemetry: Using gRPC log exporter. Value of OTEL_EXPORTER: %s, endpoint: %sziOpenTelemetry: Unknown log exporter '%s', defaulting to console. Supported: console, otlp_http, otlp_grpc)r rqr:r;r<rHrrrr3r4rrZ4opentelemetry.exporter.otlp.proto.http._log_exporterrZ4opentelemetry.exporter.otlp.proto.grpc._log_exporterrjrk)r5rrZotel_logs_exporterrrrr6r6r7r*s         zOpenTelemetry._get_log_exporterc Cs>ddlm}ddlm}m}m}td|j|j |j t |j }| |j d}|jdkr6|}||ddS|jd ksE|jd ksE|jd kr[dd lm}|||||jid }||ddS|jdkse|jdkrzdd lm}Wnty~} ztd| d} ~ ww|||||jid }||ddStd|j|}||ddS)zO Get the appropriate metric reader based on the configuration. r) Histogram)AggregationTemporalityConsoleMetricExporterPeriodicExportingMetricReaderztOpenTelemetry Logger, initializing metric reader self.OTEL_EXPORTER: %s self.OTEL_ENDPOINT: %s self.OTEL_HEADERS: %srr"i)Zexport_interval_millisr,rr)OTLPMetricExporter)r$r%Zpreferred_temporalityrrzOpenTelemetry OTLP gRPC metric exporter is not available. Install `opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`).NzlOpenTelemetry: Unknown metric exporter '%s', defaulting to console. Supported: console, otlp_http, otlp_grpc)rrZ opentelemetry.sdk.metrics.exportrrrr rqr:r;r<rHrrZ6opentelemetry.exporter.otlp.proto.http.metric_exporterrZDELTAZ6opentelemetry.exporter.otlp.proto.grpc.metric_exporterrjrk) r5rrrrrrr#rrr6r6r7r~s`           z OpenTelemetry._get_metric_readerr$ signal_typecCs|s|Shd}||vrtd|||S|d}d|}||r&|S||h}|D]}d|}||rJ|dddd|}|Sq-|dsV||}|S|d|}|S)a Normalize the endpoint URL for a specific OpenTelemetry signal type. The OTLP exporters expect endpoints to use signal-specific paths: - traces: /v1/traces - metrics: /v1/metrics - logs: /v1/logs This method ensures the endpoint has the correct path for the given signal type. Args: endpoint: The endpoint URL to normalize signal_type: The telemetry signal type ('traces', 'metrics', or 'logs') Returns: Normalized endpoint URL with the correct signal path Examples: _normalize_otel_endpoint("http://collector:4318/v1/traces", "logs") -> "http://collector:4318/v1/logs" _normalize_otel_endpoint("http://collector:4318", "traces") -> "http://collector:4318/v1/traces" _normalize_otel_endpoint("http://collector:4318/v1/logs", "metrics") -> "http://collector:4318/v1/metrics" >rrrznInvalid signal_type '%s' provided to _normalize_otel_endpoint. Valid values: %s. Returning endpoint unchanged./z/v1/rz/v1)r rkrstripendswithrsplit)r5r$r valid_signals target_pathZ other_signalsZ other_signal other_pathr6r6r7rs4       z&OpenTelemetry._normalize_otel_endpointr%cCsTi}|r(t|tr!|d}|D]}|dd\}}|||<q|St|tr(|}|S)zY Convert a string or dictionary of headers into a dictionary of headers. ,=r)r1r2splitr)r%rrrrrr6r6r7r s    z%OpenTelemetry._get_headers_dictionarylogging_payloadcs,ddlm}ddlm}m}d}d}|j}|j} t|tr%t |d}n| |}t| tr6t | d}n| | }|dur|j } |j j | |||d} |j} | duri| D]\} }|j| d| |dqY|j}|dur|D]\} }|j| d| |dqt| ||j| j|d dSdS) Nrr|rrrrequest.rz response.r)rr}rrrrrr1rrrrouterrrrrrrrrr)r5rrr}rrrrrrrmanagement_endpoint_span _request_datarr _responser6r6r7&async_management_endpoint_success_hook sN     z4OpenTelemetry.async_management_endpoint_success_hookcsddlm}ddlm}m}d}d}|j}|j} t|tr't t |d}n| |}t| tr:t t | d}n| | }|dur|j } |j j | |||d} |j} | durm| D]\} }|j| d| |dq]|j}|j| dt|d| ||j| j|d dSdS) Nrr|rrrrrrr)rr}rrrrrr1rrrrrrrrrrrr2rrr)r5rrr}rrrrrrrrrrr _exceptionr6r6r7&async_management_endpoint_failure_hookP sJ     z4OpenTelemetry.async_management_endpoint_failure_hookcCs&|jjt|||j|d|jjdS)zF Create a span for the received proxy server request. r)rrrkind)rrrrrrZSERVER)r5rr%r6r6r7)create_litellm_proxy_request_started_span s   z7OpenTelemetry.create_litellm_proxy_request_started_span)NNNNN)F)NNNN)rNNNNrf)PrBrCrDrr!r2rrX staticmethodrdr[rFr{rQrYrZrrrrrrr rrrrrrtrrrrrrrrrrrrrrr2r'r(r)rrrrrr]rprrurrrr{rrrrrrrr rrrrrrrrrrr __classcell__r6r6r]r7rHs( I> > E  0 NF  F  +V  ODS$    + : 8gTE D   8 2rH):r3 dataclassesrrtypingrrrrrr r rZlitellm._loggingr rer r Z"litellm.integrations.custom_loggerrZ*litellm.litellm_core_utils.safe_json_dumpsrZlitellm.secret_managers.mainrZlitellm.types.servicesrZlitellm.types.utilsrrrrrrrrZ _SpanExporterrrZ_Contextr_SpanrZ_TracerrkrZ!_ManagementEndpointLoggingPayloadZlitellm.proxy.proxy_serverrZ_UserAPIKeyAuthr4rrr rrrr!rHr6r6r6r7sP  $                    M