o )i(@sddlmZmZddlZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZmZddlmZmZGdd d ZGd d d ZdS) ) dataclassfieldN)CallableDictHashableListOptionalTupleUnion)LMFormatEnforcerException)CharacterLevelParserForceStopParserCharacterLevelParserConfig)TokenizerPrefixTreeTokenizerPrefixTreeNodec @sNeZdZdZdeeeeefde eegefde eeeffddZ dS)TokenEnforcerTokenizerDatazTokenEnforcerTokenizerData contains all of the preprocessing for preparing the TokenEnforcer to work with a specific tokenizer. It does some calculations, so it is recommended to reuse it for multiple TokenEnforcersregular_tokensdecoder eos_token_idcCs@||_t||_||_||_ddd|jjjD|_ dS)aA Create the tokenizer data that the TokenEnforcer needs. This can be reused for multiple TokenEnforcers if they work with the same tokenizer. :param regular_tokens: A list of tuples (token_id, token_string, is_new_word_token) for all the regular (not special) tokens in the tokenizer vocabulary. Note that token_string is expected to include leading / trailing whitespaces if relevant. :param decoder: A function that decodes a list of token ids into a string. :param eos_token_id: The token id(s) of the end-of-string token(s). css |] }t|dkr|VqdS)r N)len).0Z token_strrj/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/lmformatenforcer/tokenenforcer.py sz6TokenEnforcerTokenizerData.__init__..N) rrtokenizer_treerrjoinrootchildrenkeystokenizer_alphabet)selfrrrrrr__init__s  $z#TokenEnforcerTokenizerData.__init__N) __name__ __module__ __qualname____doc__rr intstrboolrr r#rrrrr src @seZdZdZeGdddZdedefddZde e d e e fd d Z d e d dfddZ dedede e deefddZd dde e fddZdS) TokenEnforcerzTokenEnforcer provides a token filtering mechanism, given a CharacterLevelParser and some information about the tokenizer. It is the main entry point for extending lm-format-enforcer to new inference libraries. See __init__() and get_allowed_tokens()c@sBeZdZUeed<eedZee ed<eedZ ee ed<dS)TokenEnforcer.OutputTensorStateparser)default_factoryallowed_tokenscurrent_word_tokensN) r$r%r&r __annotations__rlistr/rr(r0rrrrOutputTensorState#s r3tokenizer_datar-cCsHi|_||_|j|_|j|_|j|_|j|_i|_t|jd}||_ dS)z Create a new TokenEnforcer. :param tokenizer_data: Per tokenizer data that the token enforcer needs in order to operate. :param parser: A CharacterLevelParser that defines the allowed strings. )alphabetN) prefix_states root_parserrrrrallowed_token_cacherr!config)r"r4r-r9rrrr#)s  zTokenEnforcer.__init__token_sequencereturncCst|}|dd}||jvr|j|jS||jvr/tj|jd}||j|<||||jS|j|}|||}||j|<||||jS)a Get a list of allowed tokens, given a list of tokens that were already generated. :param token_sequence: The tokens that were already generated, and the next token will be generated for. :return: A list of token ids that are allowed to be selected next. Nr-)tupler6r/r+r3r7_compute_allowed_tokens_apply_new_characters)r"r:Z sent_tupleZprev_step_tuplestateZprev_step_state new_staterrrget_allowed_tokens:s          z TokenEnforcer.get_allowed_tokens state_tokensrAr,cCszVg}|j}|dur||jvr|j||_WdS|j}||j|jj|||jr?| t |j t r:|j n|j g|sEt d||_|durT||j|<WdSWdSty^tytjtjd|t |}td|dt |j t r|j n|j g|_YdSw)Nz+Parser reached state with no allowed tokens)levelz+Unknown LMFormatEnforcer Problem. Prefix: 'z' Terminating the parser. Please open an issue at https://github.com/noamgat/lm-format-enforcer/issues with the prefix and CharacterLevelParser parameters)r- cache_keyr8r/ shortcut_key_collect_allowed_tokensrrZcan_endextend isinstancerr2 ValueErrorr Exceptionlogging basicConfigERRORr exception)r"rDrAr/rFrGprefixrrrr?Xs0       "z%TokenEnforcer._compute_allowed_tokens tree_noder/rGcCs||j|}|j}t||}t|trU|ddkrUt |dks)J|\}} } } |j j } t | j td| | } t | j | | }|| | ||dg}|D]}||}|j|}||||dqWdS)NrZ json_freetext")rItokensZget_allowed_charactersrr set intersectionrJr>rrZjson_freetext_tokensminZ max_token_lenmaxZlookup_allowed_tokens add_characterrH)r"r-rRr/rGZallowed_charactersZrelevant_charactersZcharacters_to_explore_cur_lenmin_lenmax_lencacheZ min_remainingZmax_allowed_len characterZ next_parserZnext_tree_noderrrrHxs"      z%TokenEnforcer._collect_allowed_tokensc Cstj|jd}|d}||jjvr|g|_|jj|}n|j|g|_||j}||j}|t|d}|D]-}z |j ||_Wq9t yf} zt d|d| dt |_WYd} ~ q9d} ~ ww|S)Nr=r<zReceived an invalid character 'z+', switching to ForceStopParser (Exception:))r+r3r-rZnew_word_tokensr0Ztokens_to_strsrrrZrLrMdebugr) r"rAr:rBZ new_tokenZnew_charactersZ prev_decodedZ new_decodedr`errrr@s$   z#TokenEnforcer._apply_new_charactersN)r$r%r&r'rr3rr r#rr(rCr r?rrrrHr@rrrrr+ s" r+) dataclassesrrsystypingrrrrrr r rM exceptionsr Zcharacterlevelparserr rrZtokenizerprefixtreerrrr+rrrrs$