o ưi1C@s6dZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl Z ddlmZddlmZmZddlmZdd lmZerudd lmZdd lmZdd lmZdd lmZ ddl!m"Z#ddl$m%Z&eeefZe#Z"e&Z%e Z'eZeZneZeZ"eZ%eZ'ee(efZee(efZGddde Z)GdddeZ*dS)a Batch Rate Limiter Hook This hook implements rate limiting for batch API requests by: 1. Reading batch input files to count requests and estimate tokens at submission 2. Validating actual usage from output files when batches complete 3. Integrating with the existing parallel request limiter infrastructure ## Integration & Calling This hook is automatically registered and called by the proxy system. See BATCH_RATE_LIMITER_INTEGRATION.md for complete integration details. Quick summary: - Add to PROXY_HOOKS in litellm/proxy/hooks/__init__.py - Gets auto-instantiated on proxy startup via _add_proxy_hooks() - async_pre_call_hook() fires on POST /v1/batches (batch submission) - async_log_success_event() fires on GET /v1/batches/{id} (batch completion) ) TYPE_CHECKINGAnyDictListLiteralOptionalUnion) HTTPException) BaseModelN)verbose_proxy_logger)_get_batch_job_input_file_usage_get_file_content_as_dictionary) CustomLogger)UserAPIKeyAuth)Span)RateLimitDescriptor)RateLimitStatus)$_PROXY_MaxParallelRequestsHandler_v3)InternalUsageCache)Routerc@s"eZdZUdZeed<eed<dS)BatchFileUsagezT Internal model for batch file usage tracking, used for batch rate limiting total_tokens request_countN)__name__ __module__ __qualname____doc__int__annotations__rr]/home/app/Keep/.python/lib/python3.10/site-packages/litellm/proxy/hooks/batch_rate_limiter.pyr?s  rc@seZdZdZdedefddZddded d ed e d d f ddZ de de d ed d fddZ  d de deddee d efddZde de d efddZde dede de d eee e d ff ddZd S)!_PROXY_BatchRateLimiterz Rate limiter for batch API requests. Handles rate limiting at two points: 1. Batch submission - reads input file and reserves capacity 2. Batch completion - reads output file and adjusts for actual usage internal_usage_cacheparallel_request_limitercCs||_||_dS)a Initialize the batch rate limiter. Note: These dependencies are automatically injected by ProxyLogging._add_proxy_hooks() when this hook is registered in PROXY_HOOKS. See BATCH_RATE_LIMITER_INTEGRATION.md. Args: internal_usage_cache: Cache for storing rate limit data (auto-injected) parallel_request_limiter: Existing rate limiter to integrate with (needs custom injection) N)r"r#)selfr"r#rrr __init__Os z _PROXY_BatchRateLimiter.__init__statusr descriptorsr batch_usage limit_typereturnNc sddlm}tfddt|Dd}|r||ndddd}|}|jj} || } || d} t dd } d } |d krcd | d dd| ddd|j d| d| d| }nd | d dd| ddd|j d| d| d| }t d|t| || dd)z,Raise HTTPException for rate limit exceeded.r)datetimec3s,|]\}}|ddkr|VqdS)keyZdescriptor_keyN)get).0idr&rr ms  zB_PROXY_BatchRateLimiter._raise_rate_limit_error..N)r,value rate_limitz%Y-%m-%d %H:%M:%S UTClimit_remaining current_limitrequestszBatch rate limit exceeded for r,unknown: r4z. Batch contains z requests but only z requests remaining out of z RPM limit. Limit resets at: z tokens but only z tokens remaining out of z TPM limit. Limit resets at: i)z retry-afterrate_limit_typeZreset_at) status_codedetailheaders)r+next enumeratenow timestampr# window_size fromtimestampstrftimemaxr-rrr str)r$r&r'r(r)r+Zdescriptor_index descriptorrArCZ reset_timeZreset_time_formattedZremaining_displayr7r=rr1r _raise_rate_limit_errorasV   z/_PROXY_BatchRateLimiter._raise_rate_limit_erroruser_api_key_dictdatacs^ddlm}|jj||dddd}|jj||jddIdH}|dD]&}|d }|d } |d kr4|jn|d kr;|jnd} | | krI|||||q#g} |D]N} | d } | d}| d}|durbqN| ddur|jj | |d d}| |||j|jj d| ddur|jj | |d d}| |||j|jj dqN| r|jj | |jdIdHdSdS)z Check rate limits and increment counters by the batch amounts. Raises HTTPException if any limit would be exceeded. r)RedisPipelineIncrementOperationNF)rJrKZrpm_limit_typeZtpm_limit_typeZmodel_has_failuresT)r'parent_otel_spanZ read_onlyZstatusesr;r6r8tokensr,r4r5Zrequests_per_unit)r,r4r;)r,Zincrement_valuettlZtokens_per_unit)pipeline_operationsrM)Zlitellm.types.cachingrLr#Z_create_rate_limit_descriptorsZshould_rate_limitrMrrrIr-Zcreate_rate_limit_keysappendrCZ,async_increment_tokens_with_ttl_preservation)r$rJrKr(rLr'Zrate_limit_responser&r;r6Zrequired_capacityrPrHr,r4r5Zrpm_keyZtpm_keyrrr #_check_and_increment_batch_counterss|      z;_PROXY_BatchRateLimiter._check_and_increment_batch_countersopenaifile_idcustom_llm_provider)rSZazureZ vertex_aic sz=ddlm}||}|r|dur|j||dIdH}n tj|||dIdH}t|j}t||d}t|} t |j | dWSt yX} zt d|d t| d} ~ ww) a Count number of requests and tokens in a batch input file. Args: file_id: The file ID to read custom_llm_provider: The custom LLM provider to use for token encoding user_api_key_dict: User authentication information for file access (required for managed files) Returns: BatchFileUsage with total_tokens and request_count r)"_is_base64_encoded_unified_file_idN)rTrJrTrUrJ)Zfile_content_dictionaryrU)rrz$Error counting input file usage for r:)Z1litellm.proxy.openai_files_endpoints.common_utilsrV_fetch_managed_file_contentlitellm afile_contentr contentr lenrr Exceptionr errorrG) r$rTrUrJrVZis_managed_file file_contentZfile_content_as_dictZinput_file_usagererrr count_input_file_usagesB   z._PROXY_BatchRateLimiter.count_input_file_usagec sddlm}z ddlm}m}Wnty'}z tdt|dd}~ww|dur0td|d}|dur=td t ||sFtd |durNtd |j ||j |d IdH}|S) a Fetch file content from managed files hook. This is needed for managed files because they require proper user context to verify file ownership and access permissions. Args: file_id: The managed file ID (base64 encoded) user_api_key_dict: User authentication information Returns: HttpxBinaryResponseContent with the file content r)BaseFileEndpoints) llm_routerproxy_logging_objz)Cannot import proxy_server dependencies: z7. Managed files require proxy_server to be initialized.NzBproxy_logging_obj not available. Cannot access managed files hook.Z managed_filesz9Managed files hook not found. Cannot access managed file.z7Managed files hook is not a BaseFileEndpoints instance.z6llm_router not available. Cannot access managed files.)rTZlitellm_parent_otel_spanrc) Z*litellm.llms.base_llm.files.transformationrbZlitellm.proxy.proxy_serverrcrd ImportError ValueErrorrGZget_proxy_hook isinstancerZrM) r$rTrJrbrcrdr`Zmanaged_files_objr_rrr rX*sB    z3_PROXY_BatchRateLimiter._fetch_managed_file_contentcache call_typec s|dkrtd||StdzR|d}|s$td|WS|dd}td||j|||d Id H}td |jd |j|j|d <|j|d<|j|||dId Htd|WStynty}ztj dt |dd|WYd }~Sd }~ww)aR Pre-call hook for batch operations. Only handles batch creation (acreate_batch): - Reads input file - Counts tokens and requests - Reserves rate limit capacity via parallel_request_limiter Args: user_api_key_dict: User authentication information cache: Cache instance (not used directly) data: Request data call_type: Type of call being made Returns: Modified data dict or None Raises: HTTPException: 429 if rate limit would be exceeded Z acreate_batchzMBatch rate limiter: Not handling batch creation rate limiting for call type: z9Batch rate limiter: Handling batch creation rate limiting input_file_idz9No input_file_id in batch request, skipping rate limitingrUrSz'Counting tokens from batch input file: rWNz!Batch input file usage - Tokens: z , Requests: Z_batch_token_countZ_batch_request_count)rJrKr(z3Batch rate limit check passed, counters incrementedzError in batch rate limiting: T)exc_info) r debugr-rarrrRr r]r^rG) r$rJrhrKrirjrUr(r`rrr async_pre_call_hookgs`        z+_PROXY_BatchRateLimiter.async_pre_call_hook)rSN)rrrrrParallelRequestLimiterr%rrrGrIrrrRrrrarrXrr]rmrrrr r!Fsp  5 ] : =r!)+rtypingrrrrrrrZfastapir Zpydanticr rYZlitellm._loggingr Zlitellm.batches.batch_utilsr r Z"litellm.integrations.custom_loggerrZlitellm.proxy._typesrZopentelemetry.tracer_SpanZ/litellm.proxy.hooks.parallel_request_limiter_v3rZ_RateLimitDescriptorrZ_RateLimitStatusrZ_ParallelRequestLimiterZlitellm.proxy.utilsrZ_InternalUsageCacheZlitellm.routerrZ_RouterrnrGrr!rrrr s<$