o ưi*"@s ddlmZmZmZddlmZmZddlZddlm Z ddl m Z ddl m Zgdddddddddddddddd fd ed ed eed eedeedeedeedeedeedeedeedeedeedeedeedeef ddZddZddZdS) )FIRST_COMPLETEDThreadPoolExecutorwait)ListOptionalN) print_verbose)get_optional_params)handleriXdmodelmessages functions function_call temperaturetop_pnstream max_tokenspresence_penaltyfrequency_penalty logit_biasuserrequest_timeouttimeout max_workersc!Kst}|}g}|}d}|dddtjvr&|ddd}|ddd}|dkrJt||||||p3d|| | | | | ||d}tj||tj|d}|Sd d }t|d >}||d D]0}|D]+}| }| d ||d<i}d|vrw| d}|j tj fi||}| |q]qYWdn1swYg}|D]"}z | |Wqty} z | | WYd} ~ qd} ~ ww|S)a  Batch litellm.completion function for a given model. Args: model (str): The model to use for generating completions. messages (List, optional): List of messages to use as input for generating completions. Defaults to []. functions (List, optional): List of functions to use as input for generating completions. Defaults to []. function_call (str, optional): The function call to use as input for generating completions. Defaults to "". temperature (float, optional): The temperature parameter for generating completions. Defaults to None. top_p (float, optional): The top-p parameter for generating completions. Defaults to None. n (int, optional): The number of completions to generate. Defaults to None. stream (bool, optional): Whether to stream completions or not. Defaults to None. stop (optional): The stop parameter for generating completions. Defaults to None. max_tokens (float, optional): The maximum number of tokens to generate. Defaults to None. presence_penalty (float, optional): The presence penalty for generating completions. Defaults to None. frequency_penalty (float, optional): The frequency penalty for generating completions. Defaults to None. logit_bias (dict, optional): The logit bias for generating completions. Defaults to {}. user (str, optional): The user string for generating completions. Defaults to "". deployment_id (optional): The deployment ID for generating completions. Defaults to None. request_timeout (int, optional): The request timeout for generating completions. Defaults to None. max_workers (int,optional): The maximum number of threads to use for parallel processing. Returns: list: A list of completion results. N/rZvllmF)rrrrrrstoprrrrrr custom_llm_provider)r r custom_prompt_dictoptional_paramscss.tdt||D] }||||Vq dS)z)Yield successive n-sized chunks from lst.rN)rangelen)lstrir&T/home/app/Keep/.python/lib/python3.10/site-packages/litellm/batch_completion/main.pychunks_sz batch_completion..chunksrr rr kwargs)localssplitlitellmZ provider_listr vllm_handlerZbatch_completionsr rcopypopsubmit completionappendresult Exception)!r r rrrrrrrrrrrrZ deployment_idrrrr*argsZbatch_messagesZ completionsrr!resultsr(executorZ sub_batchZ message_listZkwargs_modifiedZoriginal_kwargsfutureexcr&r&r'batch_completion s|0&    r;c sd|vr |dd|vrl|d|di}ttdC}D]}|jtjg|Rd|i|||<q"t|fdddD]\}}|durY|WdSqBWddS1sewYdSd|vr@|d}|d|d |d i}i}tt|d}|D]'}| D] } | |vr|| || <qi||}|jtjfi|||d<q|r/t d t | t d \} } t d | | D]I}z|} | WWdStyt dt |i} |D]\} }||kr t d| q|| | <q| }t d|Yqwt dt ||sWddSWddS1s;wYdS)a Send a request to multiple language models concurrently and return the response as soon as one of the models responds. Args: *args: Variable-length positional arguments passed to the completion function. **kwargs: Additional keyword arguments: - models (str or list of str): The language models to send requests to. - Other keyword arguments to be passed to the completion function. Returns: str or None: The response from one of the language models, or None if no response is received. Note: This function utilizes a ThreadPoolExecutor to parallelize requests to multiple models. It sends requests concurrently and returns the response from the first model that responds. r modelsr)cs|dS)Nr)index)xr<r&r'sz)batch_completion_models..)keyN deploymentsZ model_listr*z waiting for next result ) return_whenz done list z3 got an exception, ignoring, removing from futuresz removing keyz new futuresz done looping through futures )r0rr#r1r-r2sorteditemsr4keysrrvaluesrr5)r6r*futuresr8r r9rBZ nested_kwargsZ deploymentrAdone_r4Z new_futuresvaluer&r?r'batch_completion_modelss       ::          ) ))rLc s(ddl}dvr ddvrd}ntdt|tr$|g}nt|ttfr0t|}ntdt|dkrsz9batch_completion_models_all_responses..z=batch_completion_models_all_responses: model request failed: )concurrent.futuresr0r5 isinstancestrlisttuple TypeErrorr#rHrr4r3r) r6r* concurrentr< responsesrHr9r4er&rNr'%batch_completion_models_all_responsessH         rY)rPrrrtypingrrr-Zlitellm._loggingrZ litellm.utilsrZllms.vllm.completionr r.rRfloatintbooldictr;rLrYr&r&r&r'st        v V