o 0 i@sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZmZddlmZmZmZddlmZddlmZmZddlmZddl mZddlmZmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%ddl&Z&dd l'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5m6Z6ddl7m8Z8m9Z9ddl1m:Z:e8rddl;Z;ddl&mZ>m?Z?ddl@mAZAmBZBe,rddlCZCe0rddlDmEZEe.oe+oe/oe-ZFeFrddlGZGddlHmIZImJZJddlKmLZLddlMmNZNmOZOddlPmQZQddlRmSZSddlTmUZUddlVmWZWmXZXmYZYmZZZm[Z[ddl\m]Z]ddl^m_Z_m`Z`maZambZbmcZcmdZdmeZemfZfmgZgmhZhmiZimjZjmkZkmlZlmmZmddlnmoZodd lpmqZqmrZrmsZsGd!d"d"eod#d$ZtGd%d&d&e]d#d$ZuGd'd(d(eSd#d$ZveretZwereuZxerevZyhd)Zzhd*Z{hd+Z|e9}e~Zd,d-d.d/iZeeZd0ZGd1d2d2ejZd3efd4d5Zd6ed7d8d9d8fd:d;ZGdd?d?ZeGd@dAdAZGdBdCdCe:Ze~dDkreZedSdS)EN)ArgumentParser Namespace)AsyncGenerator GeneratorIterable)asynccontextmanager) dataclassfield)BytesIO)Thread)Optional TypedDictUnion model_infoHF_HUB_OFFLINE) DecodeStream)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES)is_fastapi_availableis_librosa_availableis_openai_availableis_pydantic_availableis_uvicorn_availableis_vision_available) AutoConfigLogitsProcessorListPreTrainedTokenizerFastProcessorMixinTextIteratorStreamer)is_torch_availablelogging)BaseTransformersCLICommand) AutoProcessorBitsAndBytesConfigGenerationConfigPreTrainedModel)ContinuousBatchingManager RequestStatus)Image)FastAPI HTTPException)CORSMiddleware) JSONResponseStreamingResponse) Transcription)TranscriptionCreateParamsBase)ChatCompletionMessageParam)ChatCompletionChunkChoice ChoiceDeltaChoiceDeltaToolCallChoiceDeltaToolCallFunction)CompletionCreateParamsStreaming)ResponseResponseCompletedEventResponseContentPartAddedEventResponseContentPartDoneEventResponseCreatedEvent ResponseErrorResponseErrorEventResponseFailedEventResponseInProgressEventResponseOutputItemAddedEventResponseOutputItemDoneEventResponseOutputMessageResponseOutputTextResponseTextDeltaEventResponseTextDoneEvent)ResponseCreateParamsStreaming) BaseModel TypeAdapterValidationErrorc@eZdZUdZeed<dS))TransformersResponseCreateParamsStreamingz OpenAI's ResponseCreateParamsStreaming with an additional field for the generation config (as a json string). generation_configN__name__ __module__ __qualname____doc__str__annotations__rXrXi/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/transformers/commands/serving.pyrO{  rOF)totalc@rN)+TransformersCompletionCreateParamsStreamingz OpenAI's CompletionCreateParamsStreaming with additional fields for the generation config (as a json string) and passing the request_id rPNrQrXrXrXrYr\rZr\c@s.eZdZUdZeed<eed<dZeed<dS)%TransformersTranscriptionCreateParamsz OpenAI's TranscriptionCreateParamsBase with an additional field for the generation config (as a json string). filerPFstreamN) rRrSrTrUbytesrWrVr_boolrXrXrXrYr]s r]> storeZmax_tool_callsuserprevious_response_id backgroundinclude service_tiertextZ reasoning top_logprobsZ truncationprompt tool_choice>Zmax_completion_tokensnZ modalitiesrkstoprbZ function_calllogprobsZpresence_penaltyrgresponse_formatrcZreasoning_effortmetadataZstream_optionsZ functionsparallel_tool_callsZ predictionZweb_search_optionsriZaudio>Ztimestamp_granularitiesrfZchunking_strategylanguagerjroZqwenz z )startendz x-request-idc@seZdZdZdZdZdZdS)ModalityLLMVLMSTTTTSN)rRrSrTrvrwrxryrXrXrXrYrus ruargscCst|S)z~ Factory function used to instantiate serving server from provided command line arguments. Returns: ServeCommand ) ServeCommand)rzrXrXrYserve_command_factorysr|reqmodel_generation_configr(returncKsZ|ddurtd it|d}nt|}|jd i|}|D]\}}|dur3t|||q%|ddurBt |d|_ |ddurPt |d|_ |ddur^t |d|_ |ddurj|d|_ |ddurv|d|_|ddurt |d|_t |dd krd |_|d durt |d |_|d durt|d |S)a Creates a generation config from the parameters of the request. If a generation config is passed in the request, it will be used as a baseline for parameterization. Otherwise, we will use the model's default generation config. Other parameters in the request will be applied on top of the baseline. Args: req (`dict`): The request which may optionally contain generation parameters. model_generation_config (`GenerationConfig`): The model's default generation config. kwargs (`dict`): Additional parameters to set in the generation config. Returns: The prepared `GenerationConfig` object. rPNZmax_output_tokensZ max_tokensZfrequency_penaltyZ logit_biasrm temperaturegFtop_pseedrX)getr(jsonloadscopydeepcopyupdateitemssetattrintmax_new_tokensfloatZrepetition_penaltyZ sequence_biasZ stop_stringsr do_samplertorch manual_seed)r}r~kwargsrPZnon_standard_kwargskvrXrXrY!create_generation_config_from_reqs6    rc@s eZdZdZddZddZdS) ToolStatez7Lightweight class to keep track of the tool call state.cCs |dSN)resetselfrXrXrY__init__'s zToolState.__init__cCsd|_d|_d|_d|_dS)z>Reset the tool call state (assumes we're outside a tool call).FrN)inside_tool_callhas_tool_name_definedarg_nesting_levelbufferrrXrXrYr*s zToolState.resetN)rRrSrTrUrrrXrXrXrYr$s rc @sReZdZdZ ddddedeedfdd Zd d Zd d Z ddZ ddZ dS) TimedModelz A class that holds a PreTrainedModel instance and its associated processor. Automatically deletes the instances after a specified timeout. Nmodelr)timeout_seconds processor)r rcCs>||_t|j|_||_||_t|j|j|_ |j dSr) rrVZ name_or_path _name_or_pathrr threadingTimertimeout_reached_timerrs)rrrrrXrXrYr8s  zTimedModel.__init__cCs*|jt|j|j|_|jdS)z2Reset the timer for the deletion of the instances.N)rcancelrrrrrsrrXrXrY reset_timerEs zTimedModel.reset_timercCsZt|dr)|jdur+|`|`d|_d|_ttjr"tj|j dSdSdS)z>Delete the wrapped model and processor and clean up resources.rN) hasattrrrgcZcollectrcudaZ is_availableZ empty_cacherrrrXrXrY delete_modelKs  zTimedModel.delete_modelcCs&|t|jd|jddS)Nz was removed from memory after z seconds of inactivity)rloggerinforrrrXrXrYr[szTimedModel.timeout_reachedcCst|d p |jduS)z)Check if the instances have been deleted.rN)rrrrXrXrY is_deleted_szTimedModel.is_deletedr) rRrSrTrUrr rrrrrrrXrXrXrYr2s     rc@seZdZUdZedddidZeed<edddidZe ed <ed d gd d dZ e e ed<eddgd d dZ e e ed<edddidZ eed<ed ddidZe e ed<edddidZeed<edddidZeed<eddddgd dZe ed<edddidZeed<eddd idZe ed!<ed"dd#idZeed$<ed%dd&idZeed'<ed(dd)idZe ed*<ed dd+idZe eed,<eddd-idZeed.<eddd/idZeed0<ed dd1idZe e ed2<d3d4Zd S)5ServeArgumentsz Arguments for the serve CLI. See the metadata arg for each argument's description -- the metadata will be printed with `transformers serve --help` Fhelpz8Whether to use continuous batching for chat completions.)defaultrpcontinuous_batchingautozfDevice to use for inference; will default to `auto` andplace the model on an accelerator if available.deviceNzA`torch_dtype` is deprecated! Please use `dtype` argument instead.)rZbfloat16Zfloat16Zfloat32)rchoices torch_dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.dtypez2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitZnf4zQuantization type.Zfp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz$Interface the server will listen to.hosti@zPort the server will listen to.porti,z@Time in seconds after which a model will be removed from memory. model_timeoutrz8Logging level as a string. Example: 'info' or 'warning'. log_levelz1The default seed for torch, should be an integer. default_seedztWhether to enable CORS. Some apps that make requests from external domains (e.g. Cursor) require CORS to be enabled. enable_corsz+Whether to turn on strict input validation.input_validationzName of the model to be forced on all requests. This is useful for testing Apps that don't allow changing models in the request. force_modelcCsN|jdur#|jdur|j|_dS|j|jkr%td|jd|jddSdS)z(Only used for BC `torch_dtype` argument.Nz`torch_dtype` z and `dtype` zn have different values. `torch_dtype` is deprecated and will be removed in 4.59.0, please set `dtype` instead.)rr ValueErrorrrXrXrY __post_init__s    zServeArguments.__post_init__)rRrSrTrUr rrarWrrVrr rrrrrrrrrrrrrrrrrrXrXrXrYrds  rc@s$eZdZedefddZdefddZdede d d d e fd d Z defddZ defddZ defddZ        dHdedeedeedeedeedeeddeedeedefd d!Zd"d#defd$d%Zd&d'Zejdeeeeffd(d)Zd*ededeedffd+d,Zedd-defd.d/Zed0efd1d2Z d*ede!eddffd3d4Z"d*ede!eddffd5d6Z#d*ede!eddffd7d8Z$d*ede%fd9d:Z&ededed;fdedefd?d@Z(dAefdBdCZ)dAede*d-effdDdEZ+dAede*d-e,ffdFdGZ-dS)Ir{parsercCs$tf}|jd|d}|jtddS)z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments serve)dataclass_types)funcN)r add_parser set_defaultsr|)rrZ serve_parserrXrXrYregister_subcommandsz ServeCommand.register_subcommandrzcCs tstd||_|jj|_|jrAt}|jjdur'||j_t d|t }|jj|vrAt d|d|jjd|d|jj |_ |jj durSt|jj td}|tj|jjtd}|tj|jji|_d|_d|_d|_d|_dS) NzaMissing dependencies for the serving CLI. Please install with `pip install transformers[serving]`z-No attn_implementation passed, defaulting to z"Continuous batching only supports z as attn_implementation, got z#Try setting `--attn_implementation=` transformersz+transformers.generation.continuous_batching)serve_dependencies_available ImportErrorrzruse_continuous_batchingr*Z default_attention_implementationrrrZ#supported_attention_implementationsrrrrrr# get_loggersetLevelZ log_levelsrlower loaded_models#running_continuous_batching_manager last_messages last_kv_cache last_model)rrzZdefault_attn_implZsupported_attn_implZtransformers_loggerZ cb_loggerrXrXrYrs@        zServeCommand.__init__requestschema validatorrL unused_fieldsc Cstd|t|}|j}||}|r(td|tdd|d|jjriz| |Wnt yQ}ztd| td| dd}~ww||@} | rktd| tdd| ddSdS)a Validates the request against the schema, and checks for unexpected keys. Args: request (`dict`): The request to validate. schema (`TypedDict`): The schema of the request to validate. It is a `TypedDict` definition. validator (`TypeAdapter`): The validator to use to validate the request. Built from `schema`. unused_fields (`set`): Fields accepted by `schema`, but not used in `transformers serve`. Raises: HTTPException: If the request is invalid or contains unexpected or unused fields. zValidating request: z Unexpected keys in the request: i) status_codedetailzValidation error: NzUnused fields in the request: ) rdebugsetkeysZ__mutable_keys__errorr.rzrZvalidate_pythonrMerrors) rrrrrZ input_keysZ possible_keysZunexpected_keyseZunused_fields_in_requestrXrXrY_validate_requests.   zServeCommand._validate_requestcC|j|tttddSN)rrrr)rrOresponse_validatorUNUSED_RESPONSE_FIELDSrrrXrXrYvalidate_response_requestA  z&ServeCommand.validate_response_requestcCrr)rr\completion_validatorUNUSED_CHAT_COMPLETION_FIELDSrrXrXrY validate_chat_completion_requestIrz-ServeCommand.validate_chat_completion_requestcCrr)rr]transcription_validatorUNUSED_TRANSCRIPTION_FIELDSrrXrXrYvalidate_transcription_requestQrz+ServeCommand.validate_transcription_requestrN request_idcontentrrole finish_reason tool_callsr8 decode_stream tokenizerrc Csl|dur|dur|dur||j|}t|tt|tt|||dd|dgddd} d| jd d d S) a Builds a chunk of a streaming OpenAI Chat Completion response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: request_id (`str`): The request ID. content (`str`, *optional*): Content of the response from the model. model (`str`, *optional*): The model that generated the content. role (`str`, *optional*): The role of the next content, until a new role is defined. finish_reason (`str`, *optional*): The reason the generation by the model has finished. tool_calls (`list[ChoiceDeltaToolCall]`, *optional*): Data about the tool calls, when they are triggered. Returns: `str`: The built chunk, a string containing a JSON string with the payload. N)rrrr)deltaindexrrzchat.completion.chunk)idcreatedrrZsystem_fingerprintobjectdata: TZ exclude_none )stepZ _tokenizerr5rtimer6r7model_dump_json) rrrrrrrrrchunkrXrXrYbuild_chat_completion_chunkYs("  z(ServeCommand.build_chat_completion_chunkresponserKcCsd|jdddS)a Builds a event of a streaming OpenAI Response response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: response (`BaseModel`): The response to build an event from. One of the multiple OpenAI Response output types Returns: `str`: The built chunk, a string containing a JSON string with the payload. rTrr)r )rrrXrXrYbuild_response_eventsz!ServeCommand.build_response_eventc stdtffdd }t|d}jr%|jtdgddgdgdtdd d lm}| d d |d t ffdd }| dd t ffdd }| dd |ffdd }| d| dfdd}| ddd}| dd |fdd} tj|jjjjjjdd S)!a Setup and run the FastAPI server for transformers serve. Models will be loaded and unloaded automatically based on usage and a timeout. The server will expose the following endpoints: - POST /v1/chat/completions: Generates chat completions. - POST /v1/responses: Generates responses. - POST /v1/audio/transcriptions: Generates transcriptions from audio. - GET /v1/models: Lists available models for 3rd party tools. Requires FastAPI and Uvicorn to be installed. appcsBdVjD]}|q jdurjjddddSdS)NTblocktimeout)rvaluesrrrm)rrrrXrYlifespans  z"ServeCommand.run..lifespan)r*T)Z allow_originsZallow_credentialsZ allow_methodsZ allow_headerszUCORS allow origin is set to `*`. This is not recommended for production environments.r)Requestz/v1/chat/completionsrbodycs:j|djr||jj}n|}t|ddSNrtext/event-stream media_type)rr#continuous_batching_chat_completionstatergenerate_chat_completionr1)rroutputrrXrYchat_completions   z)ServeCommand.run..chat_completionz /v1/responsescs"j|d|}t|ddSr)rgenerate_responser1)rr"rrXrY responsess   z#ServeCommand.run..responsesz/v1/audio/transcriptionsc s|4IdH5}t|dIdH|dd}td|djd|djd|djddd WdIdHn 1IdHsDwYj|d  |}t |d d S) Nr^r)r^rzReceived file: z ; MIME type: z; size: z.2fz KiBrrr) formr]readrrfilename content_typesizergenerate_transcriptionr1)rr'Zparsed_requestr"rrXrYaudio_transcriptionss(  z.ServeCommand.run..audio_transcriptionsz /v1/modelscstddS)Nlist)rdata)r0get_gen_modelsrXrrXrYget_all_modelssz(ServeCommand.run..get_all_modelsz/healthcSs tddiS)Nstatusok)r0rXrXrXrY healthchecks z%ServeCommand.run..healthcheckhttpcs>|jtp tt}||j_||IdH}||jt<|Sr)headersr X_REQUEST_IDrVuuiduuid4r r)rZ call_nextrrrXrXrYget_or_set_request_ids  z/ServeCommand.run..get_or_set_request_id)rrrN)rr-rZadd_middlewarer/rZ warning_oncefastapirpostdictoptionsrZ middlewareuvicornrunrzrrr) rrrrr#r%r-r1r4r:rXrrYr@s:    "zServeCommand.runcCs6gd}tr dd|DSdd|D}dd|DS)a. This is by no means a limit to which models may be instantiated with `transformers serve`: any chat-based model working with generate can work. This is a limited list of models to ensure we have a discoverable /v1/models endpoint for third-party integrations. ) zMenlo/Jan-nanozMenlo/Jan-nano-128kzQwen/Qwen2.5-0.5B-InstructzQwen/Qwen2.5-3B-InstructzQwen/Qwen2.5-7B-InstructzQwen/Qwen2.5-14B-Instructz meta-llama/Llama-3.1-8B-Instructz meta-llama/Llama-3.2-1B-Instructz!meta-llama/Llama-3.3-70B-InstructzHuggingFaceTB/SmolVLM-Instructz!ibm-granite/granite-vision-3.2-2bzQwen/Qwen2.5-VL-7B-InstructcSs.g|]}|dtj|dddqS)r/rrrrZowned_by)datetimenow timestampsplit.0rrXrXrY s  z/ServeCommand.get_gen_models..cSsg|]}t|qSrXrrGrXrXrYrI#scSs$g|]}|jd|j|jdqS)rrB)r created_atrEZauthorrGrXrXrYrI$sr)rmodelsZ model_infosrXrXrYr0s  zServeCommand.get_gen_modelsr}c s|djk}_|r!jdur!jjdddd_\}}t|dr0|jn|t||jj j dddd jdurW|j dd _t j_ j|j|d d dd |j}fddfdd}||d|S)a' Generates an OpenAI Chat Completion using continuous batching. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. rNTrrrFZfifo)r~ eos_token_id pad_token_idZ use_cacherZ scheduler)rPZ streamingmessagespt)return_tensorsadd_generation_promptc 3sz6j|ddVj|D]#}|jtjkr&j|ddVWdSj||jd|dVqWdSty`}zt t |j |dt |dVWYd}~dSd}~ww) N assistantrrrmrr)rrrrrdata: {"error": ""}) r rZrequest_id_iterr2r+FINISHEDZgenerated_tokens ExceptionrrrVcancel_request)rrresultr)model_id_and_revisionrrrXrYstream_chat_completion^s2    zPServeCommand.continuous_batching_chat_completion..stream_chat_completioncsz't|d}jj||jd}||D] }|VtdIdHqWdStjyBj|t d|dYdSw)NF)rrrzRequest z was cancelled.) rtolistrZ add_requestrasynciosleepCancelledErrorrZrwarning)Z_inputsrrr )rPrr]rXrYcancellation_wrapperzs zNServeCommand.continuous_batching_chat_completion..cancellation_wrapperr)process_model_namerrrmload_model_and_processorrrrrPrLrMZinit_continuous_batchingrZlogit_processorrsapply_chat_templatetor)rr}rmust_discard_cacherrinputsrcrX)rPr\rr]rrYr.s<     z0ServeCommand.continuous_batching_chat_completionr)cCsB|jj}|tvrtj}|S|tvrtj}|Std|)NzUnknown modality: ) __class__rRrrrurwrrvr)rZmodel_classnamemodalityrXrXrYget_model_modalitys  zServeCommand.get_model_modalityrkc Cs~g}|D]}|dgd}|tjkrEt|dtr|d}n"t|dtr@g}|dD]}|ddkr:||dq+d|}||d<nr|tjkrt|dtr^|dd|ddnY|dD]T}|ddkrr|d|qb|ddkrd |dd vrt d d |dd }t t t |}tjd dd} | j} || jn|dd } |dd| dqb||q|S)Nrrrrtyperh )rnrhZ image_urlbase64urlz^data:image/.+;base64,rz.pngF)suffixdeleteimage)rnrq)rurv isinstancerVr.appendjoinrwresubr,openr rp b64decodetempfileNamedTemporaryFilenamesave) rNrkprocessor_inputsmessageZparsed_messageZparsed_contentrZ image_datartr^rqrXrXrY*get_processor_inputs_from_inbound_messagess@            z7ServeCommand.get_processor_inputs_from_inbound_messagescsvjjdur jj|d<|d}|dddkrdS|djk}_\}}||}dtD]}|jj d vrO|nq?|j |d| d d ddd }| j}| d d d} djj d vrxd} t|| dd} t|jd} d} |r|sj} |djd| krj} i|| | d| dfdd}|| S)a Generates an OpenAI Chat Completion using `generate`. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. NrrNrUrrRrTtoolsrO)rQrrPZ return_dicttokenizerreq_0gptossFskip_special_tokensZ skip_promptr~ input_ids)streamerrPreturn_dict_in_generatepast_key_valuesc 3sd}d}djjdvrd}d}fdd}t|d}d }z#z|t}jd d V|D]}djjdvrH|d }||7}|rV||vrUd}q7q7dur|t d krhd|_ q7|t dkr| j|dddVq7|j r|j |7_ |j std|j } | durq7| d} d|_ tt| ddd|dd} n<|d krq7d|j vrq7|j|d7_|j|d8_|jdkrd |dddd}tt|dddd} j|d| gdVq7|d krj||dVq7j|dd V|Wn#ty8} ztt| d!t| d"VWYd} ~ n d} ~ wwW|dSW|dS|w)#NFrrT<|channel|>final<|message|>cjdi|}|j_dSNrXgeneraterrrZgenerate_outputrrrXrYgenerate_with_cache zbServeCommand.generate_chat_completion..stream_chat_completion..generate_with_cachetargetrrrRrS <|return|>rsrtr)rrrrz\"name\": \"(.*?)\"r$)r~functionZ _tool_call)rrrnrz"arguments": {{}) arguments)rrrn)rrrr)rrrmrTrVrW)config architecturesrr rsrr  removesuffixstrip_TOOL_CALL_TOKENSrrrrrxsearchgroupr8r9rcountrwrFrYrrrV) r _request_id filter_cot cot_trace_endrthreadresultsZ tool_stater[Z tool_nameZtoolrgeneration_kwargsrr\rrZtool_model_familyrXrYr]s          zEServeCommand.generate_chat_completion..stream_chat_completion)rzrrdrrerlr_MODELS_WITH_TOOL_SUPPORTrrrrfrrgrr!rrPis_continuationrget_seq_lengthshape)rr}rNrhrrkrZsupported_model_familiesrirgeneration_streamerrPrseq_lenr]rXrrYr!sf        zz%ServeCommand.generate_chat_completionc sdjk}_\}tdtr6dvr)dddgng}|dddnUtdtrjdvreddddkrXdddgd}n3d}d|dd <n&d}n!tdtrdvr}dddgng}|dntd |j |d d d }| j } ddd }dj jdvrd}t||d d}tjd}d}r|sՈj} |djd| krՈj}|t|||d |dfdd} | |S)a  Generates an OpenAI Response using `generate`. Args: req (`dict`): The request to generate an OpenAI Response for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Response events. rinput instructionssystemrmrcrrrz%inputs should be a list, dict, or strTrO)rQrPrdrrFrrNrrU)riZattention_maskrrPrrc3s4d}d}djjdvrd}d}fdd}t|d}d}d}d}zzz|t} td |td | d d d ddiidggddddd d} |d7} | Vt d|td | dd d ddiidggddddd d} |d7} | Vt d||t ddddgdd} |d7} | Vt dd|||td d!gd"d#} |d7} | Vd!}|D]=}djjdvr|d$}||7}|r||vrd}d!}qqtd%d||||d!d&d'gd(}|d7} |Vqtd)d||d|d!d&d'gd*}|d7} |Vtd+d|||td |jgd"d#}|d7}|d7} |Vtd,||t ddd-d|jggd.d}|d7}|d7} |Vtd/|td | d-d d ddii|jgdgddddd0 d}|d7} |V|Wncty}zVtd1t|td2|t|d3}|d7} |Vtd4|td | d5d d ddiigdgdddtd6t|d7d8 d}|d7} |VWYd}~n d}~wwW|dSW|dS|w)9NFrrTrcrrrrrrXrYrrzTServeCommand.generate_response..stream_response..generate_with_cacherzresponse.createdZresp_Zqueuedrformatrnrhrrqrrp) rrJr2rrrhrrr"rqrkrp)rnsequence_numberrr$zresponse.in_progressZ in_progresszresponse.output_item.addedZmsg_rrR)rrnr2rr)rnr output_indexitemzresponse.content_part.addedZ output_textr)rnrh annotations)rnitem_idrr content_indexpartrzresponse.output_text.deltagX@)tokenZlogprob)rnrrrrrrnzresponse.output_text.done)rnrrrrrhrnzresponse.content_part.donezresponse.output_item.done completed)rrnr2rrrzresponse.completed) rrJr2rrrhr"rrrqrkrpz"Exception in response generation: r)rnrrzresponse.failedfailed server_error)coder) rrJr2rrrhr"rrrqrkrpr) rrrr rsr r?r;rrrCrDrFr=rGrrHrIr>rhrErr<rrwrYrrrVrArBr@)rrrrrrrrrrJZresponse_createdZresponse_in_progressZresponse_output_item_addedZresponse_content_part_addedrr[Zresponse_output_text_deltaZresponse_output_text_doneZresponse_content_part_doneZresponse_output_item_doneZresponse_completedrZ error_eventZresponse_failedrrr\r}rrrXrYstream_responses                         %z7ServeCommand.generate_response..stream_response)rdrrerurVrvr.r=rrfrgrrrrrr!rrPrrrrrZ ones_like) rr}rhrrirrrPrrrrXrrYr$sZ       czServeCommand.generate_responsec ststd||d}||\tjddd}t|jd}jj }t |d}t j ||dd\}}||dd jd jd <||dd fd d } | S)a Generates an OpenAI Transcription using the audio file. Args: req (`dict`): The request containing the audio file and model information. Returns: `Generator[str, None, None]`: A generator that yields the transcription result. z]Missing librosa dependency for audio transcription. Please install with `pip install librosa`rTrrr^)srmonorO) sampling_raterPZinput_features)rrPrc3sHjdi}j|jddd}t|d}|jddVdS)NT)rr)rhrrX)rZ batch_decode sequencesr2r )Z generated_idsZtranscription_textZ transcriptionZ audio_inputs audio_modelaudio_processorrrXrY_generate_transcriptions  zDServeCommand.generate_transcription.._generate_transcription)rrrdload_audio_model_and_processorr!rrrPZfeature_extractorrior librosaloadrgrr) rr}r\rrPZmodel_sampling_rateZ audio_bytesZ audio_array_rrXrrYr,s2 z#ServeCommand.generate_transcriptioncCsx|dp |d}d}|jdurd}n#t|jt|kr d}ntt|jD]}|j|||kr6d}nq'||_|S)aD Determines whether the current request is a continuation of the last request. In other words, if it is the same chat session. Args: req (`dict`): The request to check. Returns: `True` if the request is a continuation of the last request, `False` otherwise. rNrTNF)rrlenrange)rr}rNZreq_continues_last_messagesirXrXrYrs  zServeCommand.is_continuationr'cCs@|jrtd|j|j|j|jd}|S|jrtdd}|Sd}|S)a Returns the quantization config for the given CLI arguments. Args: args (`ServeArguments`): The serve arguments. May contain quantization settings, device, etc. Returns: `Optional[BitsAndBytesConfig]`: The quantization config. T)rZbnb_4bit_compute_dtyperZbnb_4bit_use_double_quantZbnb_4bit_quant_storage)rN)rr'rrrr)rzquantization_configrXrXrYget_quantization_configs  z$ServeCommand.get_quantization_configmodel_idcCs*|jjdur |jj}d|vr|S|dS)aR Applies the `force_model` CLI argument and canonicalizes the model name to the format "model_id@revision". If the model_id DOESN'T contain an @, it defaults to "model_id@main". Args: model_id (`str`): The model ID. Returns: `str`: The canonicalized model name to be used N@z@main)rzr)rrrXrXrYrds  zServeCommand.process_model_namer\cCs>|j}td|d|vr|dd\}}n|d}}tj|||jd}|jdvr.|jntt |j}| |}||j |d|jd}|d urK||d <t j|fi|} tt | jd } | j|fi|} t| d d d urs| |j} | jjd uo~| jjd k} | jjd uo| jjdk} | s| rd| j_td|| |fS)a Generic method to load a model and a data processor from a model ID and revision, making use of the serve CLI arguments. Args: model_id_and_revision (`str`): The model ID and revision to load. model_cls (`type[PreTrainedModel]`): The model class to load. Returns: `tuple[PreTrainedModel, Union[ProcessorMixin, PreTrainedTokenizerFast]]`: The loaded model and data processor (tokenizer, audio processor, etc.). zLoading rr$main)revisionr)rNr)rrrZ device_maprNrrZ hf_device_mapr&z Loaded model )rzrrrFr&Zfrom_pretrainedrrgetattrrrrrrrrgrrPr max_length)rr\rzrrZdata_processorrrZ model_kwargsr architecturerZhas_default_max_lengthZhas_short_max_new_tokensrXrXrY_load_model_and_data_processor.sB   z+ServeCommand._load_model_and_data_processorcCt||jvs |j|r#||\}}t||jj|d|j|<||fS|j||j|j}|j|j}||fS)a\ Loads the text model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, PreTrainedTokenizerFast]`: The loaded text model and processor. rr rrrrrzrrrr)rr\rrrXrXrYreks     z%ServeCommand.load_model_and_processorcCr)aU Loads the audio model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, ProcessorMixin]`: The loaded audio model and processor. rr)rr\rrrXrXrYrs     z+ServeCommand.load_audio_model_and_processor)rNNNNNNN).rRrSrT staticmethodrrrrr=r rrrrrrVr rr.rrr rr@ functoolscacheanyr0rrrurlrrr!r$r,rarrrdrtuplerer rrXrXrXrYr{s - 1    8_-\ -J'0=  r{__main__)r_rprrCenumrrrrrxr|rr r8argparserrcollections.abcrrr contextlibr dataclassesrr r r typingr r rZhuggingface_hubrZhuggingface_hub.constantsrZtokenizers.decodersrrZ&transformers.models.auto.modeling_autorrZtransformers.utils.import_utilsrrrrrrrrrrr r!utilsr"r#r%rr&r'r(r)Zgeneration.continuous_batchingr*r+rZPILr,rr?r;r-r.Zfastapi.middleware.corsr/Zfastapi.responsesr0r1Z openai.types.audio.transcriptionr2Z.openai.types.audio.transcription_create_paramsr3Zopenai.types.chatr4Z'openai.types.chat.chat_completion_chunkr5r6r7r8r9Z*openai.types.chat.completion_create_paramsr:Zopenai.types.responsesr;r<r=r>r?r@rArBrCrDrErFrGrHrIZ-openai.types.responses.response_create_paramsrJZpydanticrKrLrMrOr\r]rrrrrrrrRrrr.rrr7Enumrur|r=rrrrr{rr@rXrXrXrYs                D      ;2q S