o ưi@sdZddlmZddlZddlmZmZmZmZddl m Z ddl m Z ddl mZddlmZmZmZmZmZmZmZmZeZejd d gee ged d ee fd ededeede fddZejd d gee ged dddddd ee fd ededeedeedeedeedeedeede fddZejdd gee ged d ee fded ededeede f ddZejdd gee ged d ee fded ededeede f dd Z ej!dd gee ged d ee fded ededeede f d!d"Z"ejd#d gee ged d ee fded ededeede f d$d%Z#ejd&d'gee ged d ee fded ededeede f d(d)Z$ejd&d'gee ged ddddd ee fded ededeedeedeedeedeede fd*d+Z%ejd,d'gee ged d ee fded-ed ededeede f d.d/Z&ejd,d'gee ged d ee fded-ed ededeede f d0d1Z'ej!d,d'gee ged d ee fded-ed ededeede f d2d3Z(dS)4z( OpenAI Evals API endpoints - /v1/evals )OptionalN) APIRouterDependsRequestResponse)UserAPIKeyAuth)user_api_key_auth)ProxyBaseLLMRequestProcessing)CancelEvalResponseCancelRunResponseDeleteEvalResponseEvalListEvalsResponseListRunsResponseRunRunDeleteResponsez /v1/evalszOpenAI Evals API)tags dependenciesZresponse_modelZopenaifastapi_responserequestcustom_llm_provideruser_api_key_dictc sJddlm}m}m}m}m}m} m} m} m } m } m }| IdH}|r)t |ni}|dp;|jdp;|jd}|rB||d<d|vrJ||d<t|d}z=|jdid|d |d |d d d |d|d|d|d|d|dd| d| d| d| d| d|IdHWSty}z |j||||dIdHd}~ww)a Create a new evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals" -H "Authorization: Bearer your-key" -H "Content-Type: application/json" -d '{ "name": "Test Eval", "data_source_config": {"type": "file", "file_id": "file-abc123"}, "testing_criteria": {"graders": [{"type": "llm_as_judge"}]} }' ``` Returns: Eval object with id, status, timestamps, etc. r general_settings llm_router proxy_configproxy_logging_objselect_data_generator user_api_baseuser_max_tokens user_modeluser_request_timeoutuser_temperatureversionNmodelx-litellm-modelrdatarrr route_typeZ acreate_evalrrrrrr r"r!rrr#errr#litellm.proxy.proxy_serverrrrrrrrr r!r"r#bodyorjsonloadsget query_paramsheadersr base_process_llm_request Exception_handle_llm_api_exception)rrrrrrrrrrrr r!r"r#r.r'r$ processorr*r+r+e/home/app/Keep/.python/lib/python3.10/site-packages/litellm/proxy/openai_evals_endpoints/endpoints.py create_evalsr4"            r9limitafterbeforeorderorder_byc  sddlm} m} m} m} m} m}m}m}m }m }m }| IdH}|r)t |ni}d|vr7|dur7||d<d|vrC|durC||d<d|vrO|durO||d<d|vr[|dur[||d<d|vrg|durg||d<|d pw|jd pw|jd }|r~||d <d |vr||d <t|d }z=|jdid |d|d|ddd| d| d| d| d| d |d d|d|d|d|d|d|IdHWSty}z |j||| |dIdHd}~ww)a List evaluations with pagination. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl "http://localhost:4000/v1/evals?limit=10" -H "Authorization: Bearer your-key" ``` Returns: ListEvalsResponse with list of evaluations rrNr;r<r=r>r?r$r%rr&rrrr(Z alist_evalsrrrrrr r"r!rrr#r)r+r,)rrr;r<r=r>r?rrrrrrrrrr r!r"r#r.r'r$r7r*r+r+r8 list_evalszs4!            r@z/v1/evals/{eval_id}eval_idc Rddlm}m}m}m}m} m} m} m} m } m }m }| IdH}|r)t |ni}||d<|dp?|jdp?|jd}|rF||d<d|vrN||d<t|d}z=|jdid |d |d |d d d|d|d|d|d| d|dd| d|d| d| d| d|IdHWSty}z |j||||dIdHd}~ww)a Get a specific evaluation by ID. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl "http://localhost:4000/v1/evals/eval_123" -H "Authorization: Bearer your-key" ``` Returns: Eval object rrNrAr$r%rr&rrrr(Z aget_evalrrrrrr r"r!rrr#r)r+r,rArrrrrrrrrrrr r!r"r#r.r'r$r7r*r+r+r8get_evalt4            rDc rB)a Update an evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123" -H "Authorization: Bearer your-key" -H "Content-Type: application/json" -d '{"name": "Updated Name"}' ``` Returns: Updated Eval object rrNrAr$r%rr&rrrr(Z aupdate_evalrrrrrr r"r!rrr#r)r+r,rCr+r+r8 update_eval?st4            rFc rB)a Delete an evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X DELETE "http://localhost:4000/v1/evals/eval_123" -H "Authorization: Bearer your-key" ``` Returns: DeleteEvalResponse with deletion confirmation rrNrAr$r%rr&rrrr(Z adelete_evalrrrrrr r"r!rrr#r)r+r,rCr+r+r8 delete_evalrErGz/v1/evals/{eval_id}/cancelc rB)a Cancel a running evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123/cancel" -H "Authorization: Bearer your-key" ``` Returns: CancelEvalResponse with cancellation confirmation rrNrAr$r%rr&rrrr(Z acancel_evalrrrrrr r"r!rrr#r)r+r,rCr+r+r8 cancel_evalrErHz/v1/evals/{eval_id}/runszOpenAI Evals API - Runsc sxddlm}m}m}m}m} m} m} m} m } m }m }| IdH}|r)t |ni}||d<|jdpR|jdpR|dpRt|dtrQ|didnd}|rY||d<d|vra||d<t|d }z=|jdid |d |d |d dd|d|d|d|d| d|dd| d|d| d| d| d|IdHWSty}z |j||||dIdHd}~ww)a Create a new run for an evaluation. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` - Pass model via body: `{"model": "gpt-4-account-1"}` - Pass model via completion.model: `{"completion": {"model": "gpt-4-account-1"}}` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123/runs" -H "Authorization: Bearer your-key" -H "Content-Type: application/json" -d '{ "data_source": {"type": "dataset", "dataset_id": "dataset_123"}, "completion": {"model": "gpt-4", "temperature": 0.7} }' ``` Returns: Run object with id, status, timestamps, etc. rrNrAr%r$ completionrr&rrrr(Z acreate_runrrrrrr r"r!rrr#r)r+)r-rrrrrrrr r!r"r#r.r/r0r3r1r2 isinstancedictr r4r5r6rCr+r+r8 create_runYsx4#  $         rLc  sxddlm} m} m} m} m} m}m}m}m }m }m }|||||d}|j dp.|j d}|r5||d<d|vr=||d<t|d}za|jdid|d |d |d d d | d| d| d| d| d| ddurqt| dndd|d|d|d|d|d|IdHWSd|d|d|d|d|d|IdHWSty}z |j||| |dIdHd}~ww)a List all runs for an evaluation with pagination. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` Example usage: ```bash curl "http://localhost:4000/v1/evals/eval_123/runs?limit=10" -H "Authorization: Bearer your-key" ``` Returns: ListRunsResponse with list of runs rr)rAr;r<r=r>r%r$rr&rrrr(Z alist_runsrrrrrNr r"r!rrr#r)r+)r-rrrrrrrr r!r"r#r3r1r2r r4strr5r6)rArrr;r<r=r>rrrrrrrrrr r!r"r#r'r$r7r*r+r+r8 list_runss4                rNz!/v1/evals/{eval_id}/runs/{run_id}run_idc s2ddlm}m}m}m} m} m} m} m} m }m }m }||d}|j dp+|j d}|r2||d<d|vr>|dur>||d<t|d}z=|jdid |d |d |d d d| d|d|d|d| d| dd| d|d|d| d| d|IdHWSty}z |j||| |dIdHd}~ww)a Get a specific run by ID. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` Example usage: ```bash curl "http://localhost:4000/v1/evals/eval_123/runs/run_456" -H "Authorization: Bearer your-key" ``` Returns: Run object with full details rr)rArOr%r$rNr&rrrr(Zaget_runrrrrrr r"r!rrr#r)r+)r-rrrrrrrr r!r"r#r3r1r2r r4r5r6)rArOrrrrrrrrrrrr r!r"r#r'r$r7r*r+r+r8get_runsp4         rPc Zddlm}m}m}m} m} m} m} m} m }m }m }| IdH}|r)t |ni}||d<||d<|dpC|jdpC|jd}|rJ||d<d|vrR||d<t|d }z=|jdid |d |d |d dd| d|d|d|d| d|dd| d|d|d| d| d|IdHWSty}z |j||| |dIdHd}~ww)a Cancel a running run. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` Example usage: ```bash curl -X POST "http://localhost:4000/v1/evals/eval_123/runs/run_456/cancel" -H "Authorization: Bearer your-key" ``` Returns: CancelRunResponse with cancellation confirmation rrNrArOr$r%rr&rrrr(Z acancel_runrrrrrr r"r!rrr#r)r+r,rArOrrrrrrrrrrrr r!r"r#r.r'r$r7r*r+r+r8 cancel_runtv4            rSc rQ)a Delete a run. Model-based routing (for multi-account support): - Pass model via header: `x-litellm-model: gpt-4-account-1` - Pass model via query: `?model=gpt-4-account-1` Example usage: ```bash curl -X DELETE "http://localhost:4000/v1/evals/eval_123/runs/run_456" -H "Authorization: Bearer your-key" ``` Returns: RunDeleteResponse with deletion confirmation rrNrArOr$r%rr&rrrr(Z adelete_runrrrrrr r"r!rrr#r)r+r,rRr+r+r8 delete_runrTrU))__doc__typingrr/ZfastapirrrrZlitellm.proxy._typesrZ$litellm.proxy.auth.user_api_key_authrZ'litellm.proxy.common_request_processingr Zlitellm.types.llms.openai_evalsr r r r rrrrZrouterpostrMr9r1intr@rDrFdeleterGrHrLrNrPrSrUr+r+r+r8s    (   Y  c  V  X  V  Z  ]  Y  S  W