o ưi @sddlZddlZddlZddlmZmZmZmZmZddl m Z ddl Z ddl m Z m Z mZmZmZddlmZddlmZddlmZddlTdd lmZmZmZdd lmZdd lmZm Z dd l!m"Z"e Z#e#j$d e egdgde#j$de egdgde#j$de egdgde efdededefddZ%e#j$de egdgde efdededefddZ&e#j'de egdgde#j'de egdgde#j'de egdgde efde(dededefddZ)e#j*de egdgde#j*de egdgde#j*de egdgde efde(dededefd d!Z+e#j'd"e egdgde#j'd#e egdgde#j'd$e egdgde efde(dededefd%d&Z,e#j$d'e egdgde#j$d(e egdgde#j$d)e egdgde efdededefd*d+Z-e#j$d,e egdgde#j$d-e egdgde#j$d.e egdgde efde(dededefd/d0Z.e#/d e#/de j0d1d2d3e efd4ed5e(fd6d7Z1dS)8N)Any AsyncIteratorDictOptionalcast)uuid4) APIRouterDepends HTTPExceptionRequestResponse) WebSocket)verbose_proxy_logger)ModifyResponseException)*)UserAPIKeyAuthuser_api_key_authuser_api_key_auth_websocket)ProxyBaseLLMRequestProcessing)ResponseAPIUsageResponsesAPIResponse)DeleteResponseResult /v1/responses responses) dependenciestagsz /responsesz/openai/v1/responsesrequestfastapi_responseuser_api_key_dictc%sddlm}m}m}m}m}m}m} m} m } m } m } m }m }m}m}m}||dIdH}ddlm}||dd|| |dd ||d }|rdd lm}dd lm}td |d|| |d}|}|j||dIdH}t|d:id|d|d|d|d|d|d|d|d| d| d| d|d|d|d|d| d ||St|d!}z|jd:id|d|d|d"d#d| d|d|d| d| ddd|d|d|d|d| d |IdH}|dryt |t!ry|j"d$vrydd%l#m$}t%t&|| 'd&}|ry|ryzAt(|d'ip!i}|d(d} | s9t)d)|j*d*t+d+|j,|j*|d|j*d,|d-IdHtd.|j*d/|j*W|WSt+yx}!zt-d0t.|!WYd}!~!|WSd}!~!ww|WSt/y}!zB|!j0}"| j1||!|"d1IdH|!j2}#t!d2t3d,t4t55|!j6p|dt%t7d3d4|#d5gigd6t8dddd7d8}$|$WYd}!~!Sd}!~!wt+y}!z |j9|!|| |d9IdHd}!~!ww);as Follows the OpenAI Responses API spec: https://platform.openai.com/docs/api-reference/responses Supports background mode with polling_via_cache for partial response retrieval. When background=true and polling_via_cache is enabled, returns a polling_id immediately and streams the response in the background, updating Redis cache. ```bash # Normal request curl -X POST http://localhost:4000/v1/responses -H "Content-Type: application/json" -H "Authorization: Bearer sk-1234" -d '{ "model": "gpt-4o", "input": "Tell me about AI" }' # Background request with polling curl -X POST http://localhost:4000/v1/responses -H "Content-Type: application/json" -H "Authorization: Bearer sk-1234" -d '{ "model": "gpt-4o", "input": "Tell me about AI", "background": true }' ``` r)_read_request_bodygeneral_settings llm_routernative_background_modepolling_cache_ttlpolling_via_cache_enabled proxy_configproxy_logging_objredis_usage_cacheselect_data_generator user_api_baseuser_max_tokens user_modeluser_request_timeoutuser_temperatureversionrN)should_use_polling_for_request backgroundFmodel)Zbackground_moder$ redis_cacher2r!r")background_streaming_taskResponsePollingHandlerz4Starting background response with polling for model=)r4ttl) polling_id request_datar9datapolling_handlerrrrr r!r%r&r(r+r-r,r*r)r.r; route_type aresponses)ZqueuedZ in_progress)_PROXY_LiteLLMManagedFilesZ managed_filesZ_hidden_paramsmodel_idz9No model_id found in response hidden params for response z!, skipping managed object storagez+No model_id found in response hidden paramsresponse)Zunified_object_idZ file_objectZlitellm_parent_otel_spanZmodel_object_idZ file_purposerzStored background response z* in managed objects table with unified_id=z>Failed to store background response in managed objects table: )rZoriginal_exceptionr:Zresp_contenttext)typerD completed)Z input_tokensZ output_tokensZ total_tokens)idobjectZ created_atr2outputstatususageerr&r.):litellm.proxy.proxy_serverrr r!r"r#r$r%r&r'r(r)r*r+r,r-r..litellm.proxy.response_polling.polling_handlerr0getZ3litellm.proxy.response_polling.background_streamingr5r7rinfoZgenerate_polling_idZcreate_initial_stateasyncio create_taskcopyrbase_process_llm_request isinstancerrJZ,litellm_enterprise.proxy.hooks.managed_filesr@rrZget_proxy_hookgetattrwarningrG ExceptionZstore_unified_object_iderrorstrrr:Zpost_call_failure_hookmessagerinttimer2rr_handle_llm_api_exception)%rrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r;r0Zshould_use_pollingr5r7r<r9Z initial_state processorrBr@Zmanaged_files_objZ hidden_paramsrArM_dataZviolation_textZ response_objrNrNe/home/app/Keep/.python/lib/python3.10/site-packages/litellm/proxy/response_api_endpoints/endpoints.py responses_apisLH0                                rdz/cursor/chat/completionsc sddlmddlmddlm}mm}m}m }m }m }m } m } m} m} m} ddlmddlm}ddlm}||dId H}d |vrRd |vrR|d |d <t|d }fd d}zj|jd$id|d|d|ddd|d|d|d|d|dd d| d| d| d| d|d| Id H}t||r|jd }jj|jdd!||tt ||j|jd giid d d d" }|WS|WSt!y}z |j"|||| d#Id Hd }~ww)%aa Cursor-specific endpoint that accepts Responses API input format but returns chat completions format. This endpoint handles requests from Cursor IDE which sends Responses API format (`input` field) but expects chat completions format response (`choices`, `messages`, etc.). ```bash curl -X POST http://localhost:4000/cursor/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer sk-1234" -d '{ "model": "gpt-4o", "input": [{"role": "user", "content": "Hello"}] }' Responds back in chat completions format. ``` r)responses_api_bridge)CustomStreamWrapper) rasync_data_generatorr r!r%r&r)r*r+r,r-r.)!BaseResponsesAPIStreamingIterator)r) ModelResponser/Nmessagesinputr=csdt|r+jjttt|ddd}|d}||ddd|d}|||dS|||dS) at Custom generator that transforms Responses API streaming chunks to chat completion chunks. This generator is used for the cursor endpoint to convert Responses API format responses to chat completion format that Cursor IDE expects. Args: response: The streaming response (BaseResponsesAPIStreamingIterator or other) user_api_key_dict: User API key authentication dict request_data: Request data containing model, logging_obj, etc. Returns: Async generator that yields SSE-formatted chat completion chunks F)Zstreaming_responseZ sync_stream json_modelitellm_logging_objr2r3N)completion_streamr2Zcustom_llm_provider logging_obj)rBrr:)rWtransformation_handlerZget_model_response_iteratorrrr\rQ)rBrr:rnroZ streamwrapperrhrfrgrerNrccursor_data_generator<s.    z6cursor_chat_completions..cursor_data_generatorrrrr>r?r&r!r r%r(r2r+r-r,r*r)r.rmr3) r2Z raw_responseZmodel_responseror:rjZoptional_paramsZlitellm_paramsencodingZapi_keyrlrLrN)#ZBlitellm.completion_extras.litellm_responses_transformation.handlerreZ,litellm.litellm_core_utils.streaming_handlerrfrOrrgr r!r%r&r)r*r+r,r-r.Z$litellm.responses.streaming_iteratorrhlitellm.types.llms.openairZlitellm.types.utilsripoprrVrWr;rQrpZtransform_responserrrZr`)rrrrr r!r%r&r)r*r+r,r-r.rrir;rarrrBroZtransformed_responserMrNrqrccursor_chat_completionss  8    -            rvz/v1/responses/{response_id}z/responses/{response_id}z"/openai/v1/responses/{response_id} response_idc sbddlm}m}m}m}m}m} m} m} m } m } m }m }m }ddlm}||rM| s2tddd|| d}||IdH}|sKtd d |d d|S||d IdH}||d <t|d}z:|jd!id|d|d|ddd|d|d|d|d| ddd| d|d|d| d| d|IdHWSty}z |j||||d IdHd}~ww)"ai Get a response by ID. Supports both: - Polling IDs (litellm_poll_*): Returns cumulative cached content from background responses - Provider response IDs: Passes through to provider API Follows the OpenAI Responses API spec: https://platform.openai.com/docs/api-reference/responses/get ```bash # Get polling response curl -X GET http://localhost:4000/v1/responses/litellm_poll_abc123 -H "Authorization: Bearer sk-1234" # Get provider response curl -X GET http://localhost:4000/v1/responses/resp_abc123 -H "Authorization: Bearer sk-1234" ``` r rr r!r%r&r'r(r)r*r+r,r-r.r6z3Redis cache not configured. Polling requires Redis. status_codedetailr4NPolling response z not found or expiredr/rwr=rrrr>Zaget_responsesr&r!r r%r(r2r+r-r,r*r)r.rLrN)rOrr r!r%r&r'r(r)r*r+r,r-r.rPr7 is_polling_idr get_staterrVrZr`)rwrrrrr r!r%r&r'r(r)r*r+r,r-r.r7r<stater;rarMrNrNrc get_responses~<(             rc sddlm}m}m}m}m}m} m} m} m } m } m }m }m }ddlm}||rb| s2tddd|| d}||IdH}|sKtd d |d d||IdH}|r\t|d d dStddd||dIdH}||d<t|d}z:|jd%id|d|d|ddd|d|d|d|d| ddd| d|d |d!| d"| d#|IdHWSty}z |j||||d$IdHd}~ww)&a Delete a response by ID. Supports both: - Polling IDs (litellm_poll_*): Deletes from Redis cache - Provider response IDs: Passes through to provider API Follows the OpenAI Responses API spec: https://platform.openai.com/docs/api-reference/responses/delete ```bash curl -X DELETE http://localhost:4000/v1/responses/resp_abc123 -H "Authorization: Bearer sk-1234" ``` rrxr6ryRedis cache not configured.rzr}Nr~r not foundrBT)rGrHZdeletedz!Failed to delete polling responser/rwr=rrrr>Zadelete_responsesr&r!r r%r(r2r+r-r,r*r)r.rLrN)rOrr r!r%r&r'r(r)r*r+r,r-r.rPr7rr rZdelete_pollingrrrVrZr`)rwrrrrr r!r%r&r'r(r)r*r+r,r-r.r7r<rsuccessr;rarMrNrNrcdelete_response s<#             rz'/v1/responses/{response_id}/input_itemsz$/responses/{response_id}/input_itemsz./openai/v1/responses/{response_id}/input_itemsc sddlm}m}m}m}m}m} m} m} m } m } m }m }||dIdH}||d<t |d}z:|jdid|d|d |d d d |d |d|d|d| ddd| d|d| d| d| d|IdHWSty}z |j||||dIdHd}~ww)z List input items for a response.r rr r!r%r&r(r)r*r+r,r-r.r/Nrwr=rrrr>Zalist_input_itemsr&r!r r%r(r2r+r-r,r*r)r.rLrNrOrr r!r%r&r(r)r*r+r,r-r.rrVrZr`)rwrrrrr r!r%r&r(r)r*r+r,r-r.r;rarMrNrNrcget_response_input_itemss`8         rz/v1/responses/compactz/responses/compactz/openai/v1/responses/compactc sddlm}m}m}m}m}m}m} m} m } m } m } m }||dIdH}t |d}z:|jdid|d|d|d d d |d |d |d|d|ddd| d| d| d| d| d|IdHWSty|}z |j||||dIdHd}~ww)a Compact a response by running a compaction pass over a conversation. Returns encrypted, opaque items that can be used to reduce context size. Follows the OpenAI Responses API spec: https://platform.openai.com/docs/api-reference/responses/compact ```bash curl -X POST http://localhost:4000/v1/responses/compact -H "Content-Type: application/json" -H "Authorization: Bearer sk-1234" -d '{ "model": "gpt-4o", "input": [{"role": "user", "content": "Hello"}] }' ``` rrr/Nr=rrrr>Zacompact_responsesr&r!r r%r(r2r+r-r,r*r)r.rLrNr)rrrrr r!r%r&r(r)r*r+r,r-r.r;rarMrNrNrccompact_responses^8%         rz"/v1/responses/{response_id}/cancelz/responses/{response_id}/cancelz)/openai/v1/responses/{response_id}/cancelc sddlm}m}m}m}m}m} m} m} m } m } m }m }m }ddlm}||re| s2tddd|| d}||IdH}|sKtd d |d d||IdH}|r_||IdH}|Stdd d||d IdH}||d<t|d}z:|jd"id|d|d|ddd|d|d|d|d| ddd| d|d|d| d| d |IdHWSty}z |j||||d!IdHd}~ww)#a Cancel a response by ID. Supports both: - Polling IDs (litellm_poll_*): Cancels background response and updates status in Redis - Provider response IDs: Passes through to provider API Follows the OpenAI Responses API spec: https://platform.openai.com/docs/api-reference/responses/cancel ```bash # Cancel polling response curl -X POST http://localhost:4000/v1/responses/litellm_poll_abc123/cancel -H "Authorization: Bearer sk-1234" # Cancel provider response curl -X POST http://localhost:4000/v1/responses/resp_abc123/cancel -H "Authorization: Bearer sk-1234" ``` rrxr6ryrrzr}Nr~rrz!Failed to cancel polling responser/rwr=rrrr>Zacancel_responsesr&r!r r%r(r2r+r-r,r*r)r.rLrN)rOrr r!r%r&r'r(r)r*r+r,r-r.rPr7rr rZcancel_pollingrrVrZr`)rwrrrrr r!r%r&r'r(r)r*r+r,r-r.r7r<rrZ updated_stater;rarMrNrNrccancel_responses<(             r.z5The model to use for the responses WebSocket session.) description websocketr2csddlm}m}m}m}m}m}m} m} m } m } ddl m } dd|j dp)ddD}i}|r9|d|d <|jd#i|Id H|d }t|jd pQg}d dd|d}t|d}|j|_fdd}||_t|d}z|j|||| ||| | | ||dd Id H\}}WnAty}z5tdz|tddt|ddId HWn tyYnw|j dddId HWYd }~d Sd }~wwz||d<| |d|| d Id H}|Id HWd Stytd!|j dd"dId HYd Sw)$a Responses API WebSocket mode endpoint. Keeps a persistent WebSocket connection for response.create events, enabling lower-latency agentic workflows with many tool-call round trips. See: https://developers.openai.com/api/docs/guides/websocket-mode/ r) r r!r%r&r)r*r+r,r-r.) route_requestcSsg|] }|r|qSrN)strip).0prNrNrc s z0responses_websocket_endpoint..zsec-websocket-protocolr3,Z subprotocolN)r2rheadershttpPOSTr)rEmethodpathr)scopecsddS)Nz {"model": "z"})encoderNr2rNrc return_bodysz1responses_websocket_endpoint..return_bodyr=Z_aresponses_websocket) rr rr.r&r%r+r-r,r*r)r2r>z"Responses WebSocket pre-call errorr[Zpre_call_error)rEr])rEr[izPre-call error)codereasonr)r;r>r!r+zResponses WebSocket errorzInternal server errorrN)!rOr r!r%r&r)r*r+r,r-r.Zlitellm.proxy.route_llm_requestrrrQsplitacceptlistrr url_urlbodyrZ common_processing_pre_call_logicrZr exceptionZ send_textjsondumpsr\close)rr2rr r!r%r&r)r*r+r,r-r.rZrequested_protocolsZ accept_kwargsr;Z headers_listrrrZbase_llm_response_processorrmrMZllm_callrNrrcresponses_websocket_endpoints0           r)2rSrr_typingrrrrruuidrZfastapirr r r r Zstarlette.websocketsr Zlitellm._loggingrZ%litellm.integrations.custom_guardrailrZlitellm.proxy._typesZ$litellm.proxy.auth.user_api_key_authrrrZ'litellm.proxy.common_request_processingrrtrrZlitellm.types.responses.mainrZrouterpostrdrvrQr\rdeleterrrrrZQueryrrNrNrNrcsx      Z  b h 5C m