o ưi@s dZddlZddlZddlmZddlmZmZmZm Z m Z m Z ddl Z ddl Z ddlmZddlmZddlmZddlmZdd lmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&dd l'm(Z(m)Z)eZ*d Z+e)       dKd ee,efde ee,efde e,de ee,efde ee,efde ee,efde ee,efde e e-e j.fde e,defddZ/e)       dKd ee,efde ee,efde e,de ee,efde ee,efde ee,efde ee,efde e e-e j.fde e,de eeeeefffddZ0e)         dLde e1de e,de e,de e,de e,de ee,efde ee,efde e e-e j.fde e,defd d!Z2e)         dLde e1de e,de e,de e,de e,de ee,efde ee,efde e e-e j.fde e,de eeeeefffd"d#Z3e)    dMd$e,de ee,efde ee,efde e e-e j.fde e,def d%d&Z4e)    dMd$e,de ee,efde ee,efde e e-e j.fde e,de eeeeefff d'd(Z5e)       dKd$e,de e,de ee,efde ee,efde ee,efde ee,efde e e-e j.fde e,defd)d*Z6e)       dKd$e,de e,de ee,efde ee,efde ee,efde ee,efde e e-e j.fde e,de eeeeefffd+d,Z7e)    dMd$e,de ee,efde ee,efde e e-e j.fde e,def d-d.Z8e)    dMd$e,de ee,efde ee,efde e e-e j.fde e,de eeeeefff d/d0Z9e)    dMd$e,de ee,efde ee,efde e e-e j.fde e,def d1d2Z:e)    dMd$e,de ee,efde ee,efde e e-e j.fde e,de eeeeefff d3d4Z;e)       dKd$e,d5ee,efde e,de ee,efde ee,efde ee,efde ee,efde e e-e j.fde e,de"fd6d7Ze)        dNd$e,de e1de e,de e,de e,de ee,efde ee,efde e e-e j.fde e,de e!eeee!fffde,de ee,efde ee,efde e e-e j.fde e,de"fd?d@Z@e)    dMd$e,d>e,de ee,efde ee,efde e e-e j.fde e,de e"eeee"fffdAdBZAe)    dMd$e,d>e,de ee,efde ee,efde e e-e j.fde e,defdCdDZBe)    dMd$e,d>e,de ee,efde ee,efde e e-e j.fde e,de eeeeefffdEdFZCe)    dMd$e,d>e,de ee,efde ee,efde e e-e j.fde e,de#fdGdHZDe)    dMd$e,d>e,de ee,efde ee,efde e e-e j.fde e,de e#eeee#fffdIdJZEdS)Ozw Main entry point for Evals API operations Provides create, list, get, update, delete, and cancel operations for evals N)partial)Any CoroutineDictListOptionalUnion)request_timeout)Logging)BaseEvalsAPIConfig)BaseLLMHTTPHandler) CancelEvalResponseCancelRunResponseCreateEvalRequestCreateRunRequestDeleteEvalResponseEvalListEvalsParamsListEvalsResponseListRunsParamsListRunsResponseRunRunDeleteResponseUpdateEvalRequest)GenericLiteLLMParams)ProviderConfigManagerclientzhttps://api.openai.comdata_source_configtesting_criterianamemetadata extra_headers extra_query extra_bodytimeoutcustom_llm_providerreturnc  t} z?t} d| d<ttf|||||||||d | } t} t| j| }| d|IdH}t |r?|IdH}|WS|}|WSt yY}z t j d||| | dd}~ww)ae Async: Create a new evaluation Args: data_source_config: Configuration for the data source testing_criteria: List of graders for all eval runs name: Optional name for the evaluation metadata: Optional additional metadata (max 16 key-value pairs) extra_headers: Additional headers for the request extra_query: Additional query parameters extra_body: Additional body parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Eval object T acreate_eval) rrrr r!r"r#r$r%Nmodelr%Zoriginal_exceptionZcompletion_kwargs extra_kwargs) localsasyncioget_event_loopr create_eval contextvars copy_contextrunrun_in_executor iscoroutine Exceptionlitellmexception_type)rrrr r!r"r#r$r%kwargs local_varsloopfuncctxfunc_with_context init_responseresponseerAI/home/app/Keep/.python/lib/python3.10/site-packages/litellm/evals/main.pyr()L    r(c KsNt} z| d} | dd} | dddu} tdi| }|dur$d}tjt|d}|dur8td |||d }|durE||d <|rL| ||pOi}|j ||d }|j |||d }|j pct }|j|dd}| jd|d| i|dtj|||||| ||pt| | d| dd }|WSty}z tjd||| | dd}~ww)a^ Create a new evaluation Args: data_source_config: Configuration for the data source testing_criteria: List of graders for all eval runs name: Optional name for the evaluation metadata: Optional additional metadata (max 16 key-value pairs) extra_headers: Additional headers for the request extra_query: Additional query parameters extra_body: Additional body parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Eval object litellm_logging_objlitellm_call_idNr(FTopenaiproviderz!CREATE eval is not supported for )rrrheaderslitellm_params)create_requestrKrJZevals)api_baseendpointr*Zoptional_paramsrKr%rshared_session url request_bodyevals_api_provider_configr%rK logging_objr!r$ _is_asyncrrPr)rA)r,getpoprrget_provider_evals_api_configr6 LlmProviders ValueErrorupdatevalidate_environmentZtransform_create_eval_requestrMDEFAULT_OPENAI_API_BASEZget_complete_urlupdate_environment_variablesbase_llm_http_handlerZcreate_eval_handlerr r5r7)rrrr r!r"r#r$r%r8r9rDrErVrKrTrLrJrSrMrRr?r@rArArBr/ns     r/limitafterbeforeorderorder_byc  r')a Async: List all evaluations Args: limit: Number of results to return per page (max 100, default 20) after: Cursor for pagination - returns evals after this ID before: Cursor for pagination - returns evals before this ID order: Sort order ('asc' or 'desc', default 'desc') order_by: Field to sort by ('created_at' or 'updated_at', default 'created_at') extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: ListEvalsResponse object T alist_evals) rarbrcrdrer!r"r$r%Nr)) r,r-r.r list_evalsr0r1r2r3r4r5r6r7)rarbrcrdrer!r"r$r%r8r9r:r;r<r=r>r?r@rArArBrfrCrfc Kstt} z| d} | dd} | dddu} tdi| }|dur$d}tjt|d}|dur8td |i}|durB||d <|durJ||d <|durR||d <|durZ||d <|durb||d<|ri| ||pli}|j ||d}|j |||d\}}| j d|d| i|dt j|||||| ||pt| | d| dd }|WSty}z tjd||| | dd}~ww)a List all evaluations Args: limit: Number of results to return per page (max 100, default 20) after: Cursor for pagination - returns evals after this ID before: Cursor for pagination - returns evals before this ID order: Sort order ('asc' or 'desc', default 'desc') order_by: Field to sort by ('created_at' or 'updated_at', default 'created_at') extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: ListEvalsResponse object rDrENrfFTrFrGz LIST evals is not supported for rarbrcrdrerI) list_paramsrKrJrOrrP rR query_paramsrTr%rKrUr!r$rVrrPr)rA)r,rWrXrrrYr6rZr[r\r]Ztransform_list_evals_requestr_r`Zlist_evals_handlerr r5r7)rarbrcrdrer!r"r$r%r8r9rDrErVrKrTrhrJrRrjr?r@rArArBrg.s     rgeval_idc t}z;t}d|d<ttf|||||d|}t} t| j|} |d| IdH} t | r;| IdH} | WS| } | WSt yU} z t j d|| ||dd} ~ ww)ax Async: Get an evaluation by ID Args: eval_id: The ID of the evaluation to fetch extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Eval object T aget_evalrkr!r"r$r%Nr)) r,r-r.rget_evalr0r1r2r3r4r5r6r7rkr!r"r$r%r8r9r:r;r<r=r>r?r@rArArBrmD   rmc K t}zv|d}|dd}|dddu} tdi|} |dur$d}tjt|d} | dur8td ||p;i} | j | | d } | j pGt } | j || | | d \}} |j dd |id|i|d tj|| || || |pkt| |d|dd }|WSty}z tjd||||dd}~ww)aq Get an evaluation by ID Args: eval_id: The ID of the evaluation to fetch extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Eval object rDrENrmFTrFrGzGET eval is not supported for rIrkrMrKrJrkrOrrP rRrTr%rKrUr!r$rVrrPr)rA)r,rWrXrrrYr6rZr[r]rMr^Ztransform_get_eval_requestr_r`Zget_eval_handlerr r5r7rkr!r"r$r%r8r9rDrErVrKrTrJrMrRr?r@rArArBron      roc st} z>t} d|d<ttf||||||||d|} t} t| j| } | d| IdH}t |r>|IdH}|WS|}|WSt yX}z t j d||| |dd}~ww)a Async: Update an evaluation Args: eval_id: The ID of the evaluation to update name: Updated name metadata: Updated metadata extra_headers: Additional headers for the request extra_query: Additional query parameters extra_body: Additional body parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Eval object T aupdate_eval)rkrr r!r"r#r$r%Nr)) r,r-r.r update_evalr0r1r2r3r4r5r6r7)rkrr r!r"r#r$r%r8r9r:r;r<r=r>r?r@rArArBrw?sJ    rwc svt} z|d} |dd} |dddu} tdi|} |dur$d}tjt|d}|dur8td |i}|durB||d <|dur[hd fd d | D}|r[||d<|rb| ||pei}|j || d}| j pqt }|j|||| |d\}}}| jd|d| i|dtj||||| | ||pt| |d|dd }|WSty}z tjd||| |dd}~ww)a Update an evaluation Args: eval_id: The ID of the evaluation to update name: Updated name metadata: Updated metadata extra_headers: Additional headers for the request extra_query: Additional query parameters extra_body: Additional body parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Eval object rDrENrwFTrFrGz!UPDATE eval is not supported for r>Zuser_api_key_authZlitellm_parent_otel_spanZuser_api_key_user_emailZuser_api_key_user_max_budgetZ user_api_keyZrequester_ip_addressrNZuser_api_key_hashZuser_api_end_user_max_budgetZuser_api_key_org_idZuser_api_key_aliasZuser_api_key_model_max_budgetZuser_api_key_request_routeZuser_api_key_metadataZrequester_metadataZuser_api_key_budget_reset_atrJZuser_api_key_auth_metadataZlitellm_api_version user_agentZglobal_max_parallel_requestsZuser_api_key_user_idZuser_api_key_team_aliasZuser_api_key_user_spendZuser_api_key_team_spendZuser_api_key_spendZuser_api_key_end_user_idZuser_api_key_max_budgetZuser_api_key_team_max_budgetZuser_api_key_team_idcsi|] \}}|vr||qSrArA).0kvZ internal_keysrArB szupdate_eval..r rI)rkupdate_requestrMrKrJrOrrPrQr)rA)r,rWrXrrrYr6rZr[itemsr\r]rMr^Ztransform_update_eval_requestr_r`Zupdate_eval_handlerr r5r7)rkrr r!r"r#r$r%r8r9rDrErVrKrTrZfiltered_metadatarJrMrRrSr?r@rAr}rBrxs       rxc rl)a Async: Delete an evaluation Args: eval_id: The ID of the evaluation to delete extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: DeleteEvalResponse object T adelete_evalrnNr)) r,r-r.r delete_evalr0r1r2r3r4r5r6r7rprArArBr rqrc Krr)a} Delete an evaluation Args: eval_id: The ID of the evaluation to delete extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: DeleteEvalResponse object rDrENrFTrFrGz!DELETE eval is not supported for rIrsrkrOrrPrtr)rA)r,rWrXrrrYr6rZr[r]rMr^Ztransform_delete_eval_requestr_r`Zdelete_eval_handlerr r5r7rurArArBrBrvrc rl)a Async: Cancel a running evaluation Args: eval_id: The ID of the evaluation to cancel extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: CancelEvalResponse object T acancel_evalrnNr)) r,r-r.r cancel_evalr0r1r2r3r4r5r6r7rprArArBrrqrc Ks"t}zw|d}|dd}|dddu} tdi|} |dur$d}tjt|d} | dur8td ||p;i} | j | | d } | j pGt } | j || | | d \}} }|j dd |id|i|d tj|| || || |plt| |d|dd }|WSty}z tjd||||dd}~ww)a Cancel a running evaluation Args: eval_id: The ID of the evaluation to cancel extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: CancelEvalResponse object rDrENrFTrFrGz!CANCEL eval is not supported for rIrsrkrOrrPrtr)rA)r,rWrXrrrYr6rZr[r]rMr^Ztransform_cancel_eval_requestr_r`Zcancel_eval_handlerr r5r7)rkr!r"r$r%r8r9rDrErVrKrTrJrMrRrSr?r@rArArBrsn      r data_sourcec  r')ap Async: Create a new run for an evaluation Args: eval_id: The ID of the evaluation to run data_source: Data source configuration for the run (can be jsonl, completions, or responses type) name: Optional name for the run metadata: Optional additional metadata extra_headers: Additional headers for the request extra_query: Additional query parameters extra_body: Additional body parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Run object T acreate_run) rkrrr r!r"r#r$r%Nr)) r,r-r.r create_runr0r1r2r3r4r5r6r7)rkrrr r!r"r#r$r%r8r9r:r;r<r=r>r?r@rArArBr@rCrc KsNt} z| d} | dd} | dddu} tdi| }|dur$d}tjt|d}|dur8td |d |i}|durD||d <|rK| ||pNi}|j ||d }|j pZt }|j ||||d \}}| jd|d| i|dtj|||||| ||ptjddd| | d| dd }|WSty}z tjd||| | dd}~ww)a Create a new run for an evaluation Args: eval_id: The ID of the evaluation to run data_source: Data source configuration for the run (can be jsonl, completions, or responses type) name: Optional name for the run metadata: Optional additional metadata extra_headers: Additional headers for the request extra_query: Additional query parameters extra_body: Additional body parameters timeout: Request timeout (default 600s for long-running operations) custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Run object rDrENrFTrFrGz CREATE run is not supported for rrrI)rkrLrKrJrOg@g@)r$connectrrPrQr)rA)r,rWrXrrrYr6rZr[r\r]rMr^Ztransform_create_run_requestr_r`Zcreate_run_handlerhttpxTimeoutr5r7)rkrrr r!r"r#r$r%r8r9rDrErVrKrTrLrJrMrRrSr?r@rArArBrs      rc  r')a Async: List all runs for an evaluation Args: eval_id: The ID of the evaluation limit: Number of results to return per page (max 100, default 20) after: Cursor for pagination - returns runs after this ID before: Cursor for pagination - returns runs before this ID order: Sort order ('asc' or 'desc', default 'desc') extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: ListRunsResponse object T alist_runs) rkrarbrcrdr!r"r$r%Nr)) r,r-r.r list_runsr0r1r2r3r4r5r6r7)rkrarbrcrdr!r"r$r%r8r9r:r;r<r=r>r?r@rArArBrrCrc Ksnt} z| d} | dd} | dddu} tdi| }|dur$d}tjt|d}|dur8td |i}|durB||d <|durJ||d <|durR||d <|durZ||d <|ra| ||pdi}|j ||d}|j ||||d\}}| j dd|i|d| i|dt j|||||| ||pt| | d| dd }|WSty}z tjd||| | dd}~ww)a List all runs for an evaluation Args: eval_id: The ID of the evaluation limit: Number of results to return per page (max 100, default 20) after: Cursor for pagination - returns runs after this ID before: Cursor for pagination - returns runs before this ID order: Sort order ('asc' or 'desc', default 'desc') extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: ListRunsResponse object rDrENrFTrFrGzLIST runs is not supported for rarbrcrdrI)rkrhrKrJrkrOrrPrir)rA)r,rWrXrrrYr6rZr[r\r]Ztransform_list_runs_requestr_r`Zlist_runs_handlerr r5r7)rkrarbrcrdr!r"r$r%r8r9rDrErVrKrTrhrJrRrjr?r@rArArBrBs      rrun_idc t}zr?r@rArArBrF   rc Ks$t}zx|d}|dd} |dddu} tdi|} |dur$d}tjt|d} | dur8td ||p;i} | j | | d } | j pGt }| j |||| | d \}} |j d||d d| i|d tj|| || || |pmt| |d|dd }|WSty}z tjd||||dd}~ww)a Get a specific run Args: eval_id: The ID of the evaluation run_id: The ID of the run to retrieve extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: Run object rDrENrFTrFrGzGET run is not supported for rIrkrrMrKrJrkrrOrrPrtr)rA)r,rWrXrrrYr6rZr[r]rMr^Ztransform_get_run_requestr_r`Zget_run_handlerr r5r7)rkrr!r"r$r%r8r9rDrErVrKrTrJrMrRr?r@rArArBrsp       rc r)a Async: Cancel a running run Args: eval_id: The ID of the evaluation run_id: The ID of the run to cancel extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: CancelRunResponse object T acancel_runrNr)) r,r-r.r cancel_runr0r1r2r3r4r5r6r7rrArArBrXrrc K&t}zy|d}|dd} |dddu} tdi|} |dur$d}tjt|d} | dur8td ||p;i} | j | | d } | j pGt }| j |||| | d \}} }|j d||d d| i|d tj|| || || |pnt| |d|dd }|WSty}z tjd||||dd}~ww)a Cancel a running run Args: eval_id: The ID of the evaluation run_id: The ID of the run to cancel extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: CancelRunResponse object rDrENrFTrFrGz CANCEL run is not supported for rIrrrOrrPrtr)rA)r,rWrXrrrYr6rZr[r]rMr^Ztransform_cancel_run_requestr_r`Zcancel_run_handlerr r5r7rkrr!r"r$r%r8r9rDrErVrKrTrJrMrRrSr?r@rArArBrp       rc r)a Async: Delete a run Args: eval_id: The ID of the evaluation run_id: The ID of the run to delete extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: RunDeleteResponse object T adelete_runrNr)) r,r-r.r delete_runr0r1r2r3r4r5r6r7rrArArBrrrc Kr)a Delete a run Args: eval_id: The ID of the evaluation run_id: The ID of the run to delete extra_headers: Additional headers for the request extra_query: Additional query parameters timeout: Request timeout custom_llm_provider: Provider name (e.g., 'openai') **kwargs: Additional parameters Returns: RunDeleteResponse object rDrENrFTrFrGz DELETE run is not supported for rIrrrOrrPrtr)rA)r,rWrXrrrYr6rZr[r]rMr^Ztransform_delete_run_requestr_r`Zdelete_run_handlerr r5r7rrArArBr8rr)NNNNNNN) NNNNNNNNN)NNNN)NNNNNNNN)F__doc__r-r0 functoolsrtypingrrrrrrrr6Zlitellm.constantsr Z*litellm.litellm_core_utils.litellm_loggingr ZLiteLLMLoggingObjZ*litellm.llms.base_llm.evals.transformationr Z*litellm.llms.custom_httpx.llm_http_handlerr Zlitellm.types.llms.openai_evalsr rrrrrrrrrrrrZlitellm.types.routerrZ litellm.utilsrrr`r^strfloatrr(r/intrfrgrmrorwrxrrrrrrrrrrrrrrrArArArBs2      <    D   z  D  w 8 _ A  8 _ 8 d   D   w  D  v ; b ; g ;