o o i; @sddlZddlZddlmZmZmZmZmZmZddl m Z m Z ddl m Z er:ddlZddlmZmZddlmZdgZeedd fZd d ZGd d d e ZGdddZdiiddfdedeededefddZdiifdedeededefddZdS)N) TYPE_CHECKINGIteratorListOptionalTupleUnion)GenerationParametersSamplingParameters) Tokenizer)PreTrainedModelPreTrainedTokenizer)OutlinesLogitsProcessor transformers)torch.DoubleTensorr.cCszddlm}WntyGddd}Ynwzddlm}Wnty1Gddd}Ynwzddlm}WntyJGd d d }Ynwzdd lm}WntycGd d d }Ynw||||fS)zGet all the Llama tokenizer types/classes that need work-arounds. When they can't be imported, a dummy class is created. r)LlamaTokenizerc@ eZdZdS)z1get_llama_tokenizer_types..LlamaTokenizerN__name__ __module__ __qualname__rrh/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/outlines/models/transformers.pyrr)LlamaTokenizerFastc@r)z5get_llama_tokenizer_types..LlamaTokenizerFastNrrrrrr%rr)CodeLlamaTokenizerc@r)z5get_llama_tokenizer_types..CodeLlamaTokenizerNrrrrrr,rr)CodeLlamaTokenizerFastc@r)z9get_llama_tokenizer_types..CodeLlamaTokenizerFastNrrrrrr3rr)Ztransformers.models.llamar ImportErrorrZtransformers.models.code_llamarr)rrrrrrrget_llama_tokenizer_typess2    rc@seZdZdZdddZdeeeefdedfd d Z d d deefd dZ dedefddZ ddZ ddZ ddZddZdS)TransformerTokenizerz@Represents a tokenizer for models in the `transformers` library. tokenizerr cKs~||_|jj|_|jj|_|jjdur|jj|j_|j|_n |jj|_|jj|_t|jj|_|j|_ t |jt |_ dSN) r eos_token_idZ eos_token pad_token_idZ pad_tokensetZall_special_tokensZspecial_tokensZ get_vocabZ vocabulary isinstanceris_llama)selfrkwargsrrr__init__As        zTransformerTokenizer.__init__promptreturn)torch.LongTensorr+cKs2d|d<d|d<|j|fi|}|d|dfS)NTpaddingptZreturn_tensors input_idsattention_maskr)r&r)r'outputrrrencodeRszTransformerTokenizer.encode token_idsr+cCs|jj|dd}|S)NT)Zskip_special_tokens)rZ batch_decode)r&r3textrrrdecodeZszTransformerTokenizer.decodetokencCs>ddlm}|j|g}|jr||s|dkrd|S|S)Nr)SPIECE_UNDERLINEz<0x20> )Ztransformers.file_utilsr7rZconvert_tokens_to_stringr% startswith)r&r6r7stringrrrconvert_token_to_string^s z,TransformerTokenizer.convert_token_to_stringcCsJt|t|r#t|drt|dr|j|jko|j|jkS|j|jkStS)N model_namer')r$typehasattrr<r'rNotImplemented)r&otherrrr__eq__js  zTransformerTokenizer.__eq__cCsddlm}t||jS)Nr)Hasher)Zdatasets.fingerprintrBhashr)r&rBrrr__hash__ts zTransformerTokenizer.__hash__cCsd|ji}|SNrr0r&staterrr __getstate__ys z!TransformerTokenizer.__getstate__cCs||ddSrE)r(rFrrr __setstate__}sz!TransformerTokenizer.__setstate__N)rr )rrr__doc__r(rstrrrr2r5r;rArDrHrIrrrrr>s     rc@s4eZdZdZ    d$ddZ d%d d d d d eed edeeffddZ d%d d d d d eed dfddZ de e e e fde dedded e e e e e e e ff ddZde e e e fde dedded ee e e e ff ddZde e e e fde dedded ef ddZddZd&d"d#ZdS)' Transformersz"Represents a `transformers` model.modelr rr cCs||_t||_dSr )rMrr)r&rMrrrrr(szTransformers.__init__Nr.r+r/past_key_valuesr*ztorch.FloatTensorc Cszddl}Wn tytdYnwd|jkr dks#JJ|r,|dd}||j||ddd|d }Wdn1sFwY|j|jfS) aCompute a forward pass through the transformer model. Parameters ---------- input_ids The input token ids. Must be one or two dimensional. attention_mask The attention mask. Must be one or two dimensional. past_key_values A tuple of tuples containing the cached key and value tensors for each attention head. Returns ------- The computed logits and the new cached key and value tensors. rNzGThe `torch` library needs to be installed to use `transformers` models.).rPTF)r/Z return_dictZoutput_attentionsZoutput_hidden_statesrN)torchrndim unsqueezeZinference_moderMlogitsrN)r&r.r/rNrQr1rrrforwards*    zTransformers.forwardcCs,||||\}}|ddddf}||fS)N.rP)rU)r&r.r/rNrTZkv_cacheZnext_token_logitsrrr__call__szTransformers.__call__promptsgeneration_parameterslogits_processorr sampling_parametersc Cst|tr|j|g\}}n|j|\}}||jj||jjd}dt|jj j vr5|d=| ||||}|j ||fi|} t|trQ| d} || S)aGenerate text using `transformers`. Arguments --------- prompts A prompt or list of prompts. generation_parameters An instance of `GenerationParameters` that contains the prompt, the maximum number of tokens, stop sequences and seed. All the arguments to `SequenceGeneratorAdapter`'s `__cal__` method. logits_processor The logits processor to use when generating text. sampling_parameters An instance of `SamplingParameters`, a dataclass that contains the name of the sampler to use and related parameters as available in Outlines. Returns ------- The generated text r.r/r/r)r$rKrr2torMdeviceinspect signaturerU parameterskeys_get_generation_kwargs_generate_output_seqsqueeze_decode_generation) r&rWrXrYrZr.r/inputsgeneration_kwargs generated_idsrrrgenerates(      zTransformers.generatec cst|tr|j|g\}}n|j|\}}||jj||jjd}dt|jj j vr6|d=| ||||}|j ||fi|} t|trR| d} t| dD]} | d| d} || VqYdS)a Temporary stream stand-in which implements stream() signature and equivalent behaviour but isn't yielded until generation completes. TODO: implement following completion of https://github.com/huggingface/transformers/issues/30810 r[r/rrPN)r$rKrr2r\rMr]r^r_rUr`rarbrcrdrangesizeselectrSre) r&rWrXrYrZr.r/rfrgrhiZoutput_group_idsrrrstreams0     zTransformers.streamc Csddlm}m}m}t|\}} } t|\} } } }}|dur$td}| dur,|| |dur6||g}nd}||| | p>d| ||| dk| dkrJ| nd|jj|jj d }t |||jjd S) zR Conert outlines generation parameters into model.generate kwargs r)GenerationConfigLogitsProcessorListset_seedNi@Z multinomialZ beam_search) max_new_tokensZ stop_stringsnum_return_sequencestop_ptop_k temperatureZ do_sampleZ num_beamsr!r")rYgeneration_configr) rrorprq dataclassesastupleintrr!r"dict)r&rWrXrYrZrorprqrsZstop_atseedZsampler num_samplesrurvrwZlogits_processor_listrxrrrrb+s:   z#Transformers._get_generation_kwargsc Ks|d}|jjdi|d|i|}|jjjr|}n |dd|jddf}|jp,d}|dkrGt|trG|d} |jp?d} | | | d}|S)Nr.rxrrrrPr) rMriconfigZis_encoder_decodershapertr$listrkview) r&rWrfrxrgr.Z output_idsrhr~Z batch_sizertrrrrcZs"     z!Transformers._generate_output_seqrh torch.TensorcsttjdkrjgdStjdkrjStjdkr2fddttDStdj)NrrrrOcsg|] }j|qSr)rr5).0rmrhr&rr xsz3Transformers._decode_generation..z7Generated outputs aren't 1D, 2D or 3D, but instead are )lenrrr5rj TypeError)r&rhrrrrers    zTransformers._decode_generation)rMr rr r )rhr)rrrrJr(rr KVCacheTyperUrVrrKrrr rirrnr|rbrcrerrrrrLs|   2   : , /rLr<r] model_kwargstokenizer_kwargsc Cs|dus|durz ddlm}m}Wn tytdw|dur$|}|dur*|}|dur2||d<|j|fi|}|dd|j|fi|} t|| S)aInstantiate a model from the `transformers` library and its tokenizer. Parameters ---------- model_name The name of the model as listed on Hugging Face's model page. device The device(s) on which the model should be loaded. This overrides the `device_map` entry in `model_kwargs` when provided. model_kwargs A dictionary that contains the keyword arguments to pass to the `from_pretrained` method when loading the model. tokenizer_kwargs A dictionary that contains the keyword arguments to pass to the `from_pretrained` method when loading the tokenizer. Returns ------- A `TransformersModel` model instance. Nr)AutoModelForCausalLM AutoTokenizerzWThe `transformers` library needs to be installed in order to use `transformers` models.Z device_mapZ padding_sideleft)rrrrZfrom_pretrained setdefaultrL) r<r]rr model_classZtokenizer_classrrrMrrrrrs$   cCs:zddlm}Wn tytdwt|||||dS)Nr)MambaForCausalLMzaThe `mamba_ssm`, `torch` and `transformer` libraries needs to be installed in order to use Mamba.)r<r]rrr)rrr)r<r]rrrrrrmambas r)ryr^typingrrrrrrZoutlines.generate.apirr Zoutlines.models.tokenizerr rQrr r Zoutlines.processorsr __all__rrrrLrKr|rrrrrsT   *C 6