o )i- @sNdZddlZddlmZmZmZmZmZmZm Z ddl Z ddl m Z ddl mZddlmZmZddlmZmZdd lmZmZdd lmZd ddd d d e eee eeeffdedeedeeeefdedef ddZdAdededefddZdBdededefddZde eefdefdd Z d!d!d d"de eefd#ed$ed%ed&edef d'd(Z! dCd)e j"d*ed+edeefd,d-Z# dCd)e j"d*ed+edeeeffd.d/Z$d0e j"de j"fd1d2Z% dDde eefd3eedefd4d5Z&d6edeefd7d8Z'dEd9eed:eedefd;d<Z(dEd9ee)d:ee)defd=d>Z*Gd?d@d@Z+dS)FzTesting utilities. The APIs in this module are used for testing and debugging and are prone to change. Don't use them in production.N)AnyDictListOptionalTupleTypeUnion) BaseModel)_core)CompiledGrammarGrammarCompiler)Grammar_convert_schema_to_str)GrammarMatcher bitmask_dtype) TokenizerInfoT)any_whitespaceindent separators strict_modeschemarrrrreturncCst|}tj|||||S)a\Convert JSON schema string to BNF grammar string. For test purposes. Parameters ---------- schema : Union[str, Type[BaseModel], Dict[str, Any]] The schema string or Pydantic model or JSON schema dict. indent : Optional[int], default: None The number of spaces for indentation. If None, the output will be in one line. separators : Optional[Tuple[str, str]], default: None Two separators used in the schema: comma and colon. Examples: (",", ":"), (", ", ": "). If None, the default separators will be used: (",", ": ") when the indent is not None, and (", ", ": ") otherwise. strict_mode : bool, default: True Whether to use strict mode. In strict mode, the generated grammar will not allow properties and items that is not specified in the schema. This is equivalent to setting unevaluatedProperties and unevaluatedItems to false. This helps LLM to generate accurate output in the grammar-guided generation with JSON schema. Returns ------- bnf_string : str The BNF grammar string. )rr testing_json_schema_to_ebnf)rrrrrZ schema_strr\/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/xgrammar/testing.pyrs$ rregexwith_rule_namecCtj||S)asConvert a regex string to BNF grammar string. For test purposes. The regex grammar follows the syntax in JavaScript (ECMA 262). Check https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions for a tutorial. Currently the following features are not supported: 1. Backreference (\1) 2. non-capturing group, naming capture groups and assertions ((?...)) 3. Unicode character class escape (\p{...}) 4. Word boundary (\b) 5. Unicode property escapes (\p{...}) 6. Quantifier with range {x,y}. Now user can just repeat the element as a workaround. This method is primarily intended for testing and debugging purposes. Parameters ---------- regex : str The regex string to be converted. Returns ------- bnf_string : str The BNF grammar string converted from the input regex. )r r_regex_to_ebnf)rrrrrr =sr root ebnf_stringroot_rule_namecCsttj||S)aConvert a BNF grammar string to a Grammar object without normalization. For test purposes. The result grammar cannot be compiled / used in GrammarMatcher. Parameters ---------- ebnf_string : str The BNF grammar string to be converted. Returns ------- grammar : Grammar The unnormalized Grammar object converted from the input BNF grammar string. )r_create_from_handler r!_ebnf_to_grammar_no_normalization)r"r#rrrr%Xs r%grammarcKs2tg}t|dd}||}t|fddi|S)aCreate a GrammarMatcher from a grammar. The tokenizer info will be set to an empty TokenizerInfo. The result matcher can only accept strings, and cannot accept tokens. Parameters ---------- grammar : Union[Grammar, str] The grammar to create the matcher from. Can be either a Grammar object or a string containing EBNF grammar. Returns ------- matcher : GrammarMatcher The created grammar matcher. FZ cache_enabledZterminate_without_stop_tokenTrr Zcompile_grammarr)r&kwargstokenizer_infogrammar_compilercompiled_grammarrrr_get_matcher_from_grammarks  r-F) debug_print print_timerequire_termination input_strr.r/r0c Csjt|}|r t}|j||d}|r)t}td|d|d||dd|s-dS|s1dS|S) a#Check if a grammar accepts a string. For test purposes. Parameters ---------- grammar : Union[Grammar, str] The grammar to check. Can be either a Grammar object or a BNF grammar string. input_str : str The input string to check. debug_print : bool, default: False Whether to print debug information during matching. print_time : bool, default: False Whether to print timing information. Returns ------- bool True if the grammar accepts the string, False otherwise. )r.z Accepting z , result: z, time: g@@z usFT)r-time monotonic_nsZ accept_stringprintZ is_terminated) r&r1r.r/r0Zgrammar_matcherstartacceptedendrrr_is_grammar_accept_strings$r8bitmask vocab_sizeindexcCsJ|jjdkr td|jtkrtdtdtj|t |j ||S)aGet the ids of the rejected tokens from the bitmask. Mainly for debug purposes. Parameters ---------- bitmask : torch.Tensor The rejected token bitmask. Should be generated by allocate_token_bitmask and filled by fill_next_token_bitmask. Should be on CPU. index : int, default: 0 The batch index of the bitmask. For batch inference, bitmask[index] will be used. Otherwise is ignored. Returns ------- rejected_token_ids : List[int] A list of rejected token ids. cpuzbitmask should be on CPU.zbitmask should be of type .) devicetype ValueErrordtyperr r_get_masked_tokens_from_bitmaskdata_ptrlistshaper9r:r;rrrrBs  rBcCstj|t|j||S)aCheck if the bitmask is a single token bitmask. Parameters ---------- bitmask : torch.Tensor The bitmask to check. Should be on CPU. vocab_size : int The size of the vocabulary. index : int, default: 0 The index of the bitmask. Returns ------- is_single_token : bool True if the bitmask is a single token bitmask, False otherwise. token_id : int The id of the token if the bitmask is a single token bitmask, -1 otherwise. )r r_is_single_token_bitmaskrCrDrErFrrrrGsrG bool_maskcCs|tj}d|jddd}|dkr!tjjj|d|fdd}||jddd}tjddt dD|j tj dtj}||j d d }|tjS) aGet the bitmask from bool mask. If the bool mask does not align with the 32-bit block size, it will add extra 1 paddings. Parameters ---------- bool_mask : torch.Tensor The rejected token bool mask. For each element value, True means the token is allowed, while False means the token is rejected. Returns ------- bitmask : torch.Tensor The rejected token bitmask. r r)valuecSsg|]}d|>qS)r r).0irrr sz)_bool_mask_to_bitmask..)r>rA)dim) totorchZint32rEnnZ functionalpadviewZtensorranger>Zint64sum)rHZbool_mask_int32Zpad_sizeZbool_mask_viewweightsr9rrr_bool_mask_to_bitmasks  rYr*cKs6|durtg}t|dd}||}t|fi|S)akCreate a GrammarMatcher from a grammar and tokenizer info. Parameters ---------- grammar : Union[Grammar, str] The grammar to create the matcher from. Can be either a Grammar object or a string containing EBNF grammar. tokenizer_info : Optional[TokenizerInfo], default: None Information about the tokenizer to use with this grammar. If None, an empty TokenizerInfo will be created. **kwargs Additional keyword arguments to pass to the GrammarMatcher constructor. Returns ------- matcher : GrammarMatcher The created grammar matcher. NFr'r()r&r*r)r+r,rrr,_get_matcher_from_grammar_and_tokenizer_infos   rZr,cCstj|jSN)r r_get_allow_empty_rule_ids_handle)r,rrrr\r\r5r7cCrr[)r r_generate_range_regexr5r7rrrr_"r^r_cCrr[)r r_generate_float_regexr`rrrra&r^rac@s~eZdZdZededefddZededefddZededefdd Zededefd d Z ededefd d Z dS)GrammarFunctorzrA utility class for transforming grammars. These methods are called during grammar parsing. For test purposes.r&rcCttjj|jS)z'Normalize the structure of the grammar.)rr$r rgrammar_functorstructure_normalizerr]r&rrrre.z#GrammarFunctor.structure_normalizercCrc)z+Inline some rule references in the grammar.)rr$r rrd rule_inlinerr]rfrrrrh5rgzGrammarFunctor.rule_inlinercCrc)z-Fuse the byte string elements in the grammar.)rr$r rrdbyte_string_fuserr]rfrrrri<rgz GrammarFunctor.byte_string_fusercCrc)z2Eliminate the not referenced rules in the grammar.)rr$r rrddead_code_eliminatorr]rfrrrrjCrgz#GrammarFunctor.dead_code_eliminatorcCrc)z4Analyze and add lookahead assertions in the grammar.)rr$r rrdlookahead_assertion_analyzerr]rfrrrrkJrgz+GrammarFunctor.lookahead_assertion_analyzerN) __name__ __module__ __qualname____doc__ staticmethodrrerhrirjrkrrrrrb*srb)T)r!)rr[)NN),ror2typingrrrrrrrrRZpydanticr baser compilerr r r&rrZmatcherrrr*rstrboolintrr r%r-r8ZTensorrBrGrYrZr\r_floatrarbrrrrs$    *  /