o ưi@sddlZddlZddlmZddlmZddlmZddlm Z m Z ddl m Z m Z GdddejZGd d d eZGd d d ZdS) N)Optional) BaseModel) print_verbose) DualCache RedisCache)DEFAULT_IN_MEMORY_TTLDEFAULT_POLLING_INTERVALc@seZdZdZeZdS)SchedulerCacheKeyszscheduler:queueN)__name__ __module__ __qualname__queuerdefault_in_memory_ttlrrH/home/app/Keep/.python/lib/python3.10/site-packages/litellm/scheduler.pyr sr c@s&eZdZUeed<eed<eed<dS)FlowItempriority request_id model_nameN)r r r int__annotations__strrrrrrs  rc@seZdZUeed<  ddeedeefddZde fdd Z d e d e d e d e fddZde d e d dfddZd e d e d e d e fddZddZd e d e fddZde d e d dfddZdS) SchedulercacheNpolling_interval redis_cachecCs6g|_d}|dur tjj}t||d|_|pt|_dS)z_ polling_interval: float or null - frequency of polling queue. Default is 3ms. N)rr)r r rvaluerrrr)selfrrrrrr__init__szScheduler.__init__requestcsD|j|jdIdH}t||j|jf|j||jdIdHdS)Nrr r) get_queuerheapqheappushrr save_queue)rrr rrr add_request0szScheduler.add_requestidrhealth_deploymentsreturncs|j|dIdH}|std|tdt|t|dkrNtd|d||dd|krLt||j||d IdHtd |d Sd Sd S) a/ Return if request can be processed. Returns: - True: * If healthy deployments are available * OR If request at the top of queue - False: * If no healthy deployments available * AND request not at the top of queue r N-Incorrectly setup. Queue is invalid. Queue={}zlen(health_deployments): rzqueue: z , seeking id=r!z Popped id: TF)r" Exceptionformatrlenr#heappopr%rr'rr(r rrrpoll:s    zScheduler.pollrcs\|j|dIdH}fdd|D}t||j||dIdHtdd|dS)z Remove a specific request from the priority queue for a model. Used when a request times out while waiting in the queue. r Ncsg|] }|dkr|qS)r+r).0itemrrr esz,Scheduler.remove_request..r!zRemoved request_id: z from queue for model: )r"r#heapifyr%r)rrrr Zfiltered_queuerr4rremove_request_s  zScheduler.remove_requestcs>|j|dIdH}|std||dd|krdSdS)zKReturn if the id is at the top of the queue. Don't pop the value from heap.r Nr*rr+TF)r"r,r-r0rrrpeekjs zScheduler.peekcCs|jS)z$Get the status of items in the queue)r )rrrrget_queue_status|szScheduler.get_queue_statuscsZ|jdur*dtjj|}|jj|dIdH}|dus!t|ts#gSt|tr*|S|jS)z> Return a queue for that specific model group N{}:{})key)rr-r r rZasync_get_cache isinstancelist)rr _cache_keyresponserrrr"s  zScheduler.get_queuer cs6|jdurdtjj|}|jj||dIdHdS)z; Save the updated queue of the model group Nr:)r;r)rr-r r rZasync_set_cache)rr rr>rrrr%s  zScheduler.save_queue)NN)r r r rrrfloatrrrr&rr=boolr1r7r8r9r"r%rrrrrs   %  r)enumr#typingrZpydanticrZlitellmrZlitellm.caching.cachingrrZlitellm.constantsrrEnumr rrrrrrs