o [q i@s4ddlmZes devrddlmZnddlZzddlZWn e y*ddlZYnwddZ dd Z d d Z d d Z GdddeZGdddeZeeGdddeZeeGdddeZeeGdddeZeeddZddZGdddeZeeGdddeZeed d!Zddl Z ddl!Z!ddlZddl"Z"ddl#Z$dd"l%m&Z&dd#l%m'Z'd$d%Z(d&d'Z)ej*Z+ej*Z,e-ed(ej.e-ed(ej.ej/e_0ej1e_2d)D]Z3e)ee3qe(ee(ee(eeZ4eZ5dd*l6m7Z7ee"j89e:e$j;)this__repr__ __builtin__ Exception __class__ __module____name__)selfZstrthisr]/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/sentencepiece/__init__.py _swig_reprs rcfdd}|S)Ncsj|dkr |||dS|dkr|j|dSt||r/ttt||tr/|||dStd|)Nrthisownz(You cannot add instance attributes to %s)rownhasattr isinstancegetattrtypepropertyAttributeErrorrnamevaluesetrrset_instance_attrs zE_swig_setattr_nondynamic_instance_variable..set_instance_attrr)rr rrr*_swig_setattr_nondynamic_instance_variables  r!cr)Ncs6t||rtt||ts|||dStd|)Nz%You cannot add class attributes to %s)rrrrr)clsrrrrrset_class_attr)s z?_swig_setattr_nondynamic_class_variable..set_class_attrr)rr#rrr'_swig_setattr_nondynamic_class_variable(s r$cr)zlClass decorator for adding a metaclass to a SWIG wrapped class - a slimmed down version of six.add_metaclasscs|j|j|jSN)r __bases____dict__copy)r" metaclassrrwrapper3z$_swig_add_metaclass..wrapperr)r*r+rr)r_swig_add_metaclass1s r-c@seZdZdZeejZdS)_SwigNonDynamicMetazKMeta class to enforce nondynamic attributes (no new attributes) for a classN)r r __qualname____doc__r$r __setattr__rrrrr.8sr.c@seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZddZee ZeeZee ZeeZee ZeeZeeZddZddZddZeZdS)1ImmutableSentencePieceText_ImmutableSentencePiececC |jSr%rrxrrr> z:ImmutableSentencePieceText_ImmutableSentencePiece.cC |j|Sr%r4r6vrrrr7> The membership flagdoccCt|tdSr%)rZ:ImmutableSentencePieceText_ImmutableSentencePiece_swiginitZ5new_ImmutableSentencePieceText_ImmutableSentencePiecerrrr__init__Az:ImmutableSentencePieceText_ImmutableSentencePiece.__init__cC t|Sr%)rZ8ImmutableSentencePieceText_ImmutableSentencePiece__piecerArrr_pieceE z8ImmutableSentencePieceText_ImmutableSentencePiece._piececCrDr%)rZ:ImmutableSentencePieceText_ImmutableSentencePiece__surfacerArrr_surfaceHrFz:ImmutableSentencePieceText_ImmutableSentencePiece._surfacecCrDr%)rZ5ImmutableSentencePieceText_ImmutableSentencePiece__idrArrr_idKrFz5ImmutableSentencePieceText_ImmutableSentencePiece._idcCrDr%)rZ8ImmutableSentencePieceText_ImmutableSentencePiece__beginrArrr_beginNrFz8ImmutableSentencePieceText_ImmutableSentencePiece._begincCrDr%)rZ6ImmutableSentencePieceText_ImmutableSentencePiece__endrArrr_endQrFz6ImmutableSentencePieceText_ImmutableSentencePiece._endcCrDr%)rZCImmutableSentencePieceText_ImmutableSentencePiece__surface_as_bytesrArrr_surface_as_bytesTrFzCImmutableSentencePieceText_ImmutableSentencePiece._surface_as_bytescCrDr%)rZAImmutableSentencePieceText_ImmutableSentencePiece__piece_as_bytesrArrr_piece_as_bytesWrFzAImmutableSentencePieceText_ImmutableSentencePiece._piece_as_bytescCsd|j|j|j|j|jS)Nz3piece: "{}" id: {} surface: "{}" begin: {} end: {} )formatpieceidsurfacebeginendrArrr__str__bs z9ImmutableSentencePieceText_ImmutableSentencePiece.__str__cCs<|j|jko|j|jko|j|jko|j|jko|j|jkSr%)rNrOrPrQrRrotherrrr__eq__js<z8ImmutableSentencePieceText_ImmutableSentencePiece.__eq__cCs tt|Sr%)hashstrrArrr__hash__m z:ImmutableSentencePieceText_ImmutableSentencePiece.__hash__N)r r r/rrrrrBrZ8delete_ImmutableSentencePieceText_ImmutableSentencePiece__swig_destroy__rErGrHrIrJrKrLrNZpiece_as_bytesrPZsurface_as_bytesrOrQrRrSrVrYrrrrr2=s.r2c@seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZee ZeeZeeZGdddZeddZddZddZddZeZdS)ImmutableSentencePieceTextcCr3r%r4r5rrrr7vr8z#ImmutableSentencePieceText.cCr9r%r4r:rrrr7vr<r=r>cCr@r%)rZ#ImmutableSentencePieceText_swiginitZnew_ImmutableSentencePieceTextrArrrrByrCz#ImmutableSentencePieceText.__init__cCrDr%)rZ'ImmutableSentencePieceText__pieces_sizerArrr _pieces_size}rFz'ImmutableSentencePieceText._pieces_sizecC t||Sr%)rZ"ImmutableSentencePieceText__piecesrindexrrr_piecesrZz"ImmutableSentencePieceText._piecescCrDr%)rZ ImmutableSentencePieceText__textrArrr_textrFz ImmutableSentencePieceText._textcCrDr%)rZ!ImmutableSentencePieceText__scorerArrr_scorerFz!ImmutableSentencePieceText._scorecCrDr%)rZ,ImmutableSentencePieceText_SerializeAsStringrArrrSerializeAsStringrFz,ImmutableSentencePieceText.SerializeAsStringcCrDr%)rZ)ImmutableSentencePieceText__text_as_bytesrArrr_text_as_bytesrFz)ImmutableSentencePieceText._text_as_bytesc@0eZdZddZddZddZddZeZd S) z9ImmutableSentencePieceText.ImmutableSentencePieceIteratorcC||_|j|_dSr%)protor]lenrrhrrrrBzBImmutableSentencePieceText.ImmutableSentencePieceIterator.__init__cC|jSr%rirArrr__len__zAImmutableSentencePieceText.ImmutableSentencePieceIterator.__len__cjt|trfddtjD|j|j|jS|dkr"|j}|dks+|jkr/tdj |S)Ncg|]}j|qSr)rhra.0irArr zYImmutableSentencePieceText.ImmutableSentencePieceIterator.__getitem__..rzpiece index is out of range) rslicerangeristartstopstep IndexErrorrhrar_rrAr __getitem__ (  zEImmutableSentencePieceText.ImmutableSentencePieceIterator.__getitem__cCddd|DS)N cSg|] }dt|qSzpieces {{ {}}}rMrXrsr6rrrruzUImmutableSentencePieceText.ImmutableSentencePieceIterator.__str__..joinrArrrrSrCzAImmutableSentencePieceText.ImmutableSentencePieceIterator.__str__Nr r r/rBrnr}rSrrrrrImmutableSentencePieceIterator  rcCrDr%)r\rrArrrpieces z!ImmutableSentencePieceText.piecescC||kSr%rdrTrrrrVz!ImmutableSentencePieceText.__eq__cC t|Sr%rWrdrArrrrYrZz#ImmutableSentencePieceText.__hash__cCs$d|j|jddd|jDS)Nztext: "{}" score: {} {}rcSrrrrrrrrurz6ImmutableSentencePieceText.__str__..)rMtextscorerrrArrrrSs  z"ImmutableSentencePieceText.__str__N)r r r/rrrrrBrZ!delete_ImmutableSentencePieceTextr[r]rarbrcrdrerZ text_as_bytesrrrrVrYrSrrrrr\us* r\c@seZdZeddddddZeZddZej Z dd Z d d Z d d Z GdddZeddZddZddZddZeZdS)ImmutableNBestSentencePieceTextcCr3r%r4r5rrrr7r8z(ImmutableNBestSentencePieceText.cCr9r%r4r:rrrr7r<r=r>cCr@r%)rZ(ImmutableNBestSentencePieceText_swiginitZ#new_ImmutableNBestSentencePieceTextrArrrrBrCz(ImmutableNBestSentencePieceText.__init__cCrDr%)rZ,ImmutableNBestSentencePieceText__nbests_sizerArrr _nbests_sizerFz,ImmutableNBestSentencePieceText._nbests_sizecCr^r%)rZ'ImmutableNBestSentencePieceText__nbestsr_rrr_nbestsrZz'ImmutableNBestSentencePieceText._nbestscCrDr%)rZ1ImmutableNBestSentencePieceText_SerializeAsStringrArrrrdrFz1ImmutableNBestSentencePieceText.SerializeAsStringc@rf) zBImmutableNBestSentencePieceText.ImmutableSentencePieceTextIteratorcCrgr%)rhrrirjrrrrBrkzKImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__init__cCrlr%rmrArrrrnrozJImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__len__crp)Ncrqr)rhrrrrArrrurvzbImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__getitem__..rznbests index is out of range) rrwrxriryrzr{r|rhrr_rrArr}r~zNImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__getitem__cCr)NrcSrznbests {{ {}}}rrrrrrurz^ImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__str__..rrArrrrSrCzJImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__str__Nrrrrr"ImmutableSentencePieceTextIteratorrrcCrDr%)rrrArrrnbestsrz&ImmutableNBestSentencePieceText.nbestscCrr%rrTrrrrVrz&ImmutableNBestSentencePieceText.__eq__cCrr%rrArrrrYrZz(ImmutableNBestSentencePieceText.__hash__cCsddd|jDS)NrcSrrrrrrrrurz;ImmutableNBestSentencePieceText.__str__..)rrrArrrrSr,z'ImmutableNBestSentencePieceText.__str__N)r r r/rrrrrBrZ&delete_ImmutableNBestSentencePieceTextr[rrrdrrrVrYrSrrrrrs rc @seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Z d4d5Z!d6d7Z"d8d9Z#d:d;Z$dd?Z&d@dAZ'dBdCZ(dDdEZ)dFdGZ*dHdIZ+dJdKZ,dLdMZ-dNdOZ.dPdQZ/dRdSZ0dTdUZ1dVdWZ2dXdYZ3dZd[Z4d\d]Z5d^d_Z6d`daZ7dbdcZ8dddeZ9dfdgZ:dhdiZ;djdkZdpdqZ?drdsZ@dtduZAdvdwZBdxdxeCdydydydydydzd{dzf d|d}ZD x x x x x x x x xdd~dZEddZFddZGddZHddZIdddZJdddZKdddZLdddZM x x x x x xdddZNdddZOdddZPdddZQdddZR x x x x x x x x xdddZSdddZTdddZUdddZVdddZWeXdxfddZYeXfddZZeXfddZ[dddZ\dddZ]dddZ^dddZ_dddZdddZ`ddZaddZbddZcddZdddZedd„ZfddĄZgdddƄZhdxS)SentencePieceProcessorcCr3r%r4r5rrrr7r8zSentencePieceProcessor.cCr9r%r4r:rrrr7r<r=r>cCr@r%)rZSentencePieceProcessor_swiginitZnew_SentencePieceProcessorrArrrrBrCzSentencePieceProcessor.__init__cCr^r%)rZ.SentencePieceProcessor_LoadFromSerializedProtorZ serializedrrrLoadFromSerializedProtorZz.SentencePieceProcessor.LoadFromSerializedProtocCr^r%)rZ,SentencePieceProcessor_SetEncodeExtraOptionsrZ extra_optionrrrSetEncodeExtraOptionsrZz,SentencePieceProcessor.SetEncodeExtraOptionscCr^r%)rZ,SentencePieceProcessor_SetDecodeExtraOptionsrrrrSetDecodeExtraOptionsrZz,SentencePieceProcessor.SetDecodeExtraOptionscCr^r%)rZ$SentencePieceProcessor_SetVocabulary)rZ valid_vocabrrr SetVocabularyrZz$SentencePieceProcessor.SetVocabularycCrDr%)rZ&SentencePieceProcessor_ResetVocabularyrArrrResetVocabulary rFz&SentencePieceProcessor.ResetVocabularycCt|||Sr%)rZ%SentencePieceProcessor_LoadVocabulary)rfilename thresholdrrrLoadVocabularyz%SentencePieceProcessor.LoadVocabularycGstj|g|RSr%)rZ'SentencePieceProcessor_CalculateEntropyrargsrrrCalculateEntropysz'SentencePieceProcessor.CalculateEntropycCrDr%)rZ#SentencePieceProcessor_GetPieceSizerArrr GetPieceSizerFz#SentencePieceProcessor.GetPieceSizecCr^r%)rZ SentencePieceProcessor_PieceToIdrrNrrr PieceToIdrZz SentencePieceProcessor.PieceToIdcCr^r%)rZ SentencePieceProcessor_IdToPiecerrOrrr IdToPiecerZz SentencePieceProcessor.IdToPiececCr^r%)rZSentencePieceProcessor_GetScorerrrrGetScorerZzSentencePieceProcessor.GetScorecCr^r%)rZ SentencePieceProcessor_IsUnknownrrrr IsUnknown rZz SentencePieceProcessor.IsUnknowncCr^r%)rZ SentencePieceProcessor_IsControlrrrr IsControl#rZz SentencePieceProcessor.IsControlcCr^r%)rZSentencePieceProcessor_IsUnusedrrrrIsUnused&rZzSentencePieceProcessor.IsUnusedcCr^r%)rZSentencePieceProcessor_IsByterrrrIsByte)rZzSentencePieceProcessor.IsBytecCrDr%)rZSentencePieceProcessor_unk_idrArrrunk_id,rFzSentencePieceProcessor.unk_idcCrDr%)rZSentencePieceProcessor_bos_idrArrrbos_id/rFzSentencePieceProcessor.bos_idcCrDr%)rZSentencePieceProcessor_eos_idrArrreos_id2rFzSentencePieceProcessor.eos_idcCrDr%)rZSentencePieceProcessor_pad_idrArrrpad_id5rFzSentencePieceProcessor.pad_idcCrDr%)rZ-SentencePieceProcessor_serialized_model_protorArrrserialized_model_proto8rFz-SentencePieceProcessor.serialized_model_protocCr^r%)rZ#SentencePieceProcessor_LoadFromFilerargrrr LoadFromFile;rZz#SentencePieceProcessor.LoadFromFilec Ct||||||||| Sr%)rZ#SentencePieceProcessor__EncodeAsIds rrenable_sampling nbest_sizealphaadd_bosadd_eosreverseemit_unk_piecerrr _EncodeAsIds>z#SentencePieceProcessor._EncodeAsIdsc Crr%)rZ&SentencePieceProcessor__EncodeAsPiecesrrrr_EncodeAsPiecesArz&SentencePieceProcessor._EncodeAsPiecesc Crr%)rZ/SentencePieceProcessor__EncodeAsSerializedProtorrrr_EncodeAsSerializedProtoDrz/SentencePieceProcessor._EncodeAsSerializedProtoc Crr%)rZ.SentencePieceProcessor__EncodeAsImmutableProtorrrr_EncodeAsImmutableProtoGrz.SentencePieceProcessor._EncodeAsImmutableProtoc Ct|||||||||| Sr%)rZ(SentencePieceProcessor__EncodeAsIdsBatch rins num_threadsrrrrrrrrrr_EncodeAsIdsBatchJz(SentencePieceProcessor._EncodeAsIdsBatchc Crr%)rZ+SentencePieceProcessor__EncodeAsPiecesBatchrrrr_EncodeAsPiecesBatchMrz+SentencePieceProcessor._EncodeAsPiecesBatchc Crr%)rZ4SentencePieceProcessor__EncodeAsSerializedProtoBatchrrrr_EncodeAsSerializedProtoBatchPrz4SentencePieceProcessor._EncodeAsSerializedProtoBatchc Crr%)rZ3SentencePieceProcessor__EncodeAsImmutableProtoBatchrrrr_EncodeAsImmutableProtoBatchSrz3SentencePieceProcessor._EncodeAsImmutableProtoBatchcCr^r%)rZ!SentencePieceProcessor__DecodeIdsridsrrr _DecodeIdsVrZz!SentencePieceProcessor._DecodeIdscCr^r%)rZ(SentencePieceProcessor__DecodeIdsAsBytesrrrr_DecodeIdsAsBytesYrZz(SentencePieceProcessor._DecodeIdsAsBytescCr^r%)rZ$SentencePieceProcessor__DecodePiecesrrrrr _DecodePieces\rZz$SentencePieceProcessor._DecodePiecescCr^r%)rZ2SentencePieceProcessor__DecodeIdsAsSerializedProtorrrr_DecodeIdsAsSerializedProto_rZz2SentencePieceProcessor._DecodeIdsAsSerializedProtocCr^r%)rZ5SentencePieceProcessor__DecodePiecesAsSerializedProtorrrr_DecodePiecesAsSerializedProtobrZz5SentencePieceProcessor._DecodePiecesAsSerializedProtocCr^r%)rZ1SentencePieceProcessor__DecodeIdsAsImmutableProtorrrr_DecodeIdsAsImmutableProtoerZz1SentencePieceProcessor._DecodeIdsAsImmutableProtocCr^r%)rZ4SentencePieceProcessor__DecodePiecesAsImmutableProtorrrr_DecodePiecesAsImmutableProtohrZz4SentencePieceProcessor._DecodePiecesAsImmutableProtocCrr%)rZ&SentencePieceProcessor__DecodeIdsBatchrrrrrr_DecodeIdsBatchkrz&SentencePieceProcessor._DecodeIdsBatchcCrr%)rZ-SentencePieceProcessor__DecodeIdsAsBytesBatchrrrr_DecodeIdsAsBytesBatchnrz-SentencePieceProcessor._DecodeIdsAsBytesBatchcCrr%)rZ7SentencePieceProcessor__DecodeIdsAsSerializedProtoBatchrrrr _DecodeIdsAsSerializedProtoBatchqrz7SentencePieceProcessor._DecodeIdsAsSerializedProtoBatchcCrr%)rZ6SentencePieceProcessor__DecodeIdsAsImmutableProtoBatchrrrr_DecodeIdsAsImmutableProtoBatchtrz6SentencePieceProcessor._DecodeIdsAsImmutableProtoBatchcCrr%)rZ)SentencePieceProcessor__DecodePiecesBatchrrrr_DecodePiecesBatchwrz)SentencePieceProcessor._DecodePiecesBatchcCrr%)rZ:SentencePieceProcessor__DecodePiecesAsSerializedProtoBatchrrrr#_DecodePiecesAsSerializedProtoBatchzrz:SentencePieceProcessor._DecodePiecesAsSerializedProtoBatchcCrr%)rZ9SentencePieceProcessor__DecodePiecesAsImmutableProtoBatchrrrr"_DecodePiecesAsImmutableProtoBatch}rz9SentencePieceProcessor._DecodePiecesAsImmutableProtoBatchc Ct|||||||Sr%)rZ(SentencePieceProcessor__NBestEncodeAsIdsrrrrrrrrrr_NBestEncodeAsIdsr,z(SentencePieceProcessor._NBestEncodeAsIdsc Crr%)rZ+SentencePieceProcessor__NBestEncodeAsPiecesrrrr_NBestEncodeAsPiecesr,z+SentencePieceProcessor._NBestEncodeAsPiecesc Crr%)rZ4SentencePieceProcessor__NBestEncodeAsSerializedProtorrrr_NBestEncodeAsSerializedProtor,z4SentencePieceProcessor._NBestEncodeAsSerializedProtoc Crr%)rZ3SentencePieceProcessor__NBestEncodeAsImmutableProtorrrr_NBestEncodeAsImmutableProtor,z3SentencePieceProcessor._NBestEncodeAsImmutableProtoc Crr%)rZ1SentencePieceProcessor__SampleEncodeAndScoreAsIds rr num_samplesrwor include_bestrrrrrrr_SampleEncodeAndScoreAsIdsrz1SentencePieceProcessor._SampleEncodeAndScoreAsIdsc Crr%)rZ4SentencePieceProcessor__SampleEncodeAndScoreAsPiecesrrrr_SampleEncodeAndScoreAsPiecesrz4SentencePieceProcessor._SampleEncodeAndScoreAsPiecesc Crr%)rZ=SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProtorrrr&_SampleEncodeAndScoreAsSerializedProtorz=SentencePieceProcessor._SampleEncodeAndScoreAsSerializedProtoc Crr%)rZ to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Soothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. num_threads: number of threads in batch processing (Default = -1, auto-detected) ) model_file model_protoN) $_sentencepiece_processor_init_native _out_type_add_bos_add_eos_reverse_emit_unk_piece_enable_sampling _nbest_size_alpha _num_threadsLoad) rrrout_typerrrrrrrrrrrInits"zSentencePieceProcessor.Initc Cs|dur|j}|dur|j}|dur|j}|dur|j}|dur#|j}|dur*|j}|dur1|j}| dur8|j} | dur?|j} |dkrW|dusS|dksS|dksS| durWt d| dusat | t uret dt |t ur|t ur|| || ||| |||| S|tur||| ||| |||| S|dks|dkr||| ||| |||| S|d kr||| ||| |||| S|t ur||||| ||||S|tur||||| ||||S|dks|dkr||||| ||||S|d kr||||| ||||St d |) a~Encode text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str. add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Soothing parameter for unigram sampling, and merge probability for BPE-dropout (probablity 'p' in BPE-dropout paper). num_threads: the number of threads used in the batch processing (Default = -1). NTrraWhen enable_sampling is True, We must specify "nbest_size > 1" or "nbest_size = -1", and "alpha". "nbest_size" is enabled only on unigram mode ignored in BPE-dropout. when "nbest_size = -1" , this method samples from all candidates on the lattice instead of nbest segmentations.num_threads must be intserialized_protorhimmutable_protozunknown out_type={})rrrrrrrrr RuntimeErrorrintlistrrXrrrrrrrrM) rinputrrrrrrrrrrrrEncodesv                  zSentencePieceProcessor.EncodecK|jd|td|SNrrrrrXrrkwargsrrrEncodeAsPieces5r,z%SentencePieceProcessor.EncodeAsPiecescKrrrr rrrr EncodeAsIds9r,z"SentencePieceProcessor.EncodeAsIdscK|jd|dd|S)Nr rrrrrrrEncodeAsSerializedProto=r,z.SentencePieceProcessor.EncodeAsSerializedProtocKr)Nr rrrrrrrEncodeAsImmutableProtoAr,z-SentencePieceProcessor.EncodeAsImmutableProtocK|jd|||tdd|SNTrrrrrrrrrrrrrrrSampleEncodeAsPiecesE z+SentencePieceProcessor.SampleEncodeAsPiecescKrrrr!rrrSampleEncodeAsIdsJr#z(SentencePieceProcessor.SampleEncodeAsIdscK|jd|||ddd|S)Nr Tr rrr!rrrSampleEncodeAsSerializedProtoOr#z4SentencePieceProcessor.SampleEncodeAsSerializedProtocKr%)Nr Tr rrr!rrrSampleEncodeAsImmutableProtoTr#z3SentencePieceProcessor.SampleEncodeAsImmutableProtocsdurjdurjdurjdurjdur#jdur*jdkr0dfddt|turKfdd|DS|S)aNBestEncode text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str. add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: nbest size Nrrcstur|Stur|Sdks$dkr.|Sdkr<|Std)Nr rhr zunknown out_type)r rrXrrrr r)rrrrrrrrr_encode}s"z3SentencePieceProcessor.NBestEncode.._encodecg|]}|qSrrrsnr)rrruz6SentencePieceProcessor.NBestEncode..)rrrrrrrr)rrrrrrrrr)r)rrrrrrrr NBestEncodeYs$ z"SentencePieceProcessor.NBestEncodecK|jd||td|SNrrrr)r/rXrrrrrrrNBestEncodeAsPieces z*SentencePieceProcessor.NBestEncodeAsPiecescKr0r1)r/r r3rrrNBestEncodeAsIdsr5z'SentencePieceProcessor.NBestEncodeAsIdscK|jd||dd|S)Nr r2rr/r3rrrNBestEncodeAsSerializedProtor5z3SentencePieceProcessor.NBestEncodeAsSerializedProtocKr7)Nr r2rr8r3rrrNBestEncodeAsImmutableProtor5z2SentencePieceProcessor.NBestEncodeAsImmutableProtoc sdur jdur jdur jdur jdur# jdur)ddur/d dur5d dur;ddkrCtdrK sKtd f dd t|turifd d |DS|S) a SampleEncodeAndScore text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str or 'serialized_proto' or 'immutable_proto' add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) num_samples: How many samples to return (Default = 1) alpha: inverse temperature for sampling wor: whether to sample without replacement (Default = false) include_best: whether to include the best tokenization, requires wor=True (Default = false) Nrg?Frznum_examples must be positivez8When include_best is True, We must specify "wor = True".c stur|  Stur"|  Sdks*dkr7|  SdkrH|  Std)Nr rhr zunknown output type)r rrXrrrr r() rrrrrrrrrrrrr)s"z._encodecr*rrr+r-rrrur.z?SentencePieceProcessor.SampleEncodeAndScore..)rrrrrr rr) rrrrrrrrrrrr) r)rrrrrrrrrrrSampleEncodeAndScores4 z+SentencePieceProcessor.SampleEncodeAndScorecK|jd|||td|SNrrrrr)r;rXrrrrrrrrSampleEncodeAndScoreAsPieces z3SentencePieceProcessor.SampleEncodeAndScoreAsPiecescKr<r=)r;r r?rrrSampleEncodeAndScoreAsIdsrAz0SentencePieceProcessor.SampleEncodeAndScoreAsIdscK|jd|||dd|S)Nr r>rr;r?rrr%SampleEncodeAndScoreAsSerializedProtorAzrrDr?rrr$SampleEncodeAndScoreAsImmutableProtorAz;SentencePieceProcessor.SampleEncodeAndScoreAsImmutableProtocCs|dur|j}|dust|turtd|sdS|turt|tur)||gSt|tur5||gSt|turt|dksIt|dturN||St|dtur[||St|dturt|ddksut|ddtur{| ||St|ddtur| ||S|t urt|tur| |gSt|tur||gSt|turt|dkst|dtur| |St|dtur||St|dturt|ddkst|ddtur| ||St|ddtur| ||S|dkrzt|tur||gSt|tur||gSt|turzt|dks3t|dtur8||St|dturF||St|dturzt|ddksct|ddturi|||St|ddturz|||S|dkrt|tur||gSt|tur||gSt|turt|dkst|dtur||St|dtur||St|dturt|ddkst|ddtur|||St|ddtur|||Std)zDecode processed id or token sequences. Args: out_type: output type. str, bytes or 'serialized_proto' or 'immutable_proto' (Default = str) num_threads: the number of threads used in the batch processing (Default = -1). Nr rrr r zunknown output or input type)rrr r rXrrrrirrbytesrrrrrrrrrr)rrrrrrrDecodes       $         $        (        (  zSentencePieceProcessor.DecodecK|jd||d|SrrHrrrrrrr DecodePiecesgr,z#SentencePieceProcessor.DecodePiecescKrIrrJrKrrr DecodeIdskr,z SentencePieceProcessor.DecodeIdsr cKrIrrJrKrrrDecodePiecesAsSerializedProtoor,z4SentencePieceProcessor.DecodePiecesAsSerializedProtocKrIrrJrKrrrDecodeIdsAsSerializedProtosr,z1SentencePieceProcessor.DecodeIdsAsSerializedProtor cKrIrrJrKrrrDecodePiecesAsImmutableProtowr,z3SentencePieceProcessor.DecodePiecesAsImmutableProtocKrIrrJrKrrrDecodeIdsAsImmutableProto{r,z0SentencePieceProcessor.DecodeIdsAsImmutableProtocCsPt|tur"|dur |j}|dust|turtd||||S|||S)zCalculate sentence entropyNr )rrrr r rr)rrrrrrrrs  c4fddt|turfdd|DS|S)Ncr|S|Sr%rrr(r with_offsetsrr _normalize  z4SentencePieceProcessor.Normalize.._normalizecr*rrrrWrrrur.z4SentencePieceProcessor.Normalize..rrrrrVrrWrrVr Normalize z SentencePieceProcessor.NormalizecKs,i}|D] \}}t|||<q||Sr%)itemsrXr)rr new_kwargskeyrrrrOverrideNormalizerSpecs z-SentencePieceProcessor.OverrideNormalizerSpeccC|Sr%rrArrr piece_sizez!SentencePieceProcessor.piece_sizecCrcr%rdrArrr vocab_sizerfz!SentencePieceProcessor.vocab_sizecCrcr%rrArrr __getstate__rfz#SentencePieceProcessor.__getstate__cC|||dSr%rBrrrrrr __setstate__z#SentencePieceProcessor.__setstate__cCrcr%rdrArrrrnrfzSentencePieceProcessor.__len__cCs ||Sr%)rrrrrr}rFz"SentencePieceProcessor.__getitem__cCs(|r|rtd|r||S||S)aOverwride SentencePieceProcessor.Load to support both model_file and model_proto. Args: model_file: The sentencepiece model file path. model_proto: The sentencepiece model serialized proto. Either `model_file` or `model_proto` must be set. z-model_file and model_proto must be exclusive.)r rr)rrrrrrrs   zSentencePieceProcessor.Load) NNNNNNNNNNN)NNNNNNr%)r )r )ir r r/rrrrrBrZdelete_SentencePieceProcessorr[rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr rrrrrrr"r$r&r'r/r4r6r9r:r;r@rBrErFrXrHrLrMrNrOrPrQr]rbrergrirmrnr}rrrrrrs  2 ^     :    M    _        rcCrDr%)rSetRandomGeneratorSeed)seedrrrrprFrpcCrDr%)rSetMinLogLevel)r;rrrrrrFrrc@seZdZeddddddZddZeZedd Z ed d Z ed d Z eddZ eddZ edddZedddZdS)SentencePieceTrainercCr3r%r4r5rrrr7r8zSentencePieceTrainer.cCr9r%r4r:rrrr7r<r=r>cOstd)NzNo constructor defined)r)rrrrrrrBrfzSentencePieceTrainer.__init__cCrDr%)rZ%SentencePieceTrainer__TrainFromString)rrrr_TrainFromStringrz%SentencePieceTrainer._TrainFromStringcCrDr%)rZ"SentencePieceTrainer__TrainFromMaprrrr _TrainFromMaprz"SentencePieceTrainer._TrainFromMapcCr^r%)rZ#SentencePieceTrainer__TrainFromMap2riterrrr_TrainFromMap2 z#SentencePieceTrainer._TrainFromMap2cCrDr%)rZ#SentencePieceTrainer__TrainFromMap3rurrr_TrainFromMap3rz#SentencePieceTrainer._TrainFromMap3cCr^r%)rZ#SentencePieceTrainer__TrainFromMap4rwrrr_TrainFromMap4rzz#SentencePieceTrainer._TrainFromMap4Nc Ks|durt|turt|Sdd}d}d}i}|D]\}}|dvr(|}q|dvr/|}q||||<q|rM|rAt||}nt|}||dS|rUt||St |S)zDTrain Sentencepiece model. Accept both kwargs and legacy string arg.NcSsZt|tur)tjddkrt}nt}tj|dd}|dd|D| St |S)zEncode value to CSV..rr)lineterminatorcSsg|]}t|qSr)rX)rsr;rrrrur.z@SentencePieceTrainer._Train.._encode..) rrsysrStringIOBytesIOcsvwriterwriterowgetvaluerX)rfrrrrr)s z,SentencePieceTrainer._Train.._encode)sentence_iteratorZsentence_reader) model_writer) rrXrsrtr_r|r{writeryrv) rrr)rrr`rarrrrr_Trains.      zSentencePieceTrainer._TraincKsDt|dtjdd|i|WddS1swYdS)N)ostreamrr) _LogStreamrsr)rZ logstreamrrrrTrains "zSentencePieceTrainer.Trainr%ro)r r r/rrrBrr staticmethodrtrvryr{r|rrrrrrrss$      ,rsc@seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZddZddZ       d"ddZd#ddZddZd d!ZdS)$SentencePieceNormalizercCr3r%r4r5rrrr7r8z SentencePieceNormalizer.cCr9r%r4r:rrrr7r<r=r>cCr@r%)rZ SentencePieceNormalizer_swiginitZnew_SentencePieceNormalizerrArrrrB rCz SentencePieceNormalizer.__init__cCr^r%)rZ/SentencePieceNormalizer_LoadFromSerializedProtorrrrr$rZz/SentencePieceNormalizer.LoadFromSerializedProtocCr^r%)rZ'SentencePieceNormalizer_LoadFromRuleTSV)rrrrrLoadFromRuleTSV'rZz'SentencePieceNormalizer.LoadFromRuleTSVcCr^r%)rZ(SentencePieceNormalizer_LoadFromRuleName)rrrrrLoadFromRuleName*rZz(SentencePieceNormalizer.LoadFromRuleNamecCrDr%)rZ.SentencePieceNormalizer_serialized_model_protorArrrr-rFz.SentencePieceNormalizer.serialized_model_protocCr^r%)rZ$SentencePieceNormalizer_LoadFromFilerrrrr0rZz$SentencePieceNormalizer.LoadFromFilecCr^r%)rZ"SentencePieceNormalizer__Normalizerrrrr3rZz"SentencePieceNormalizer._NormalizecCr^r%)rZ-SentencePieceNormalizer__NormalizeWithOffsetsrrrrr6rZz-SentencePieceNormalizer._NormalizeWithOffsetscCrr%)rZ&SentencePieceNormalizer__SetProtoFieldrrrr_SetProtoField9rz&SentencePieceNormalizer._SetProtoFieldNFc Cst||r ||}n|r||}n|r||}n |r$||}ntd|r>|d||d||d|dSdS)aInitialzie sentencePieceNormalizer. Args: model_file: The sentencepiece model file path. model_proto: The sentencepiece model serialized proto. rule_tsv: The normalization rule file in TSV format. rule_name: Pre-defined normalization name. add_dummy_prefix: add dummy prefix. escape_whitespaces: escape whitespaces. remove_extra_whitespaces: remove extra whitespaces. zno model is specifiedadd_dummy_prefixescape_whitespacesremove_extra_whitespacesN)%_sentencepiece_normalizer_init_nativerrrrr r) rrrZrule_tsvZ rule_namerrrstatusrrrr<s      zSentencePieceNormalizer.InitcrR)NcrSr%rTr(rUrrrWcrXz5SentencePieceNormalizer.Normalize.._normalizecr*rrrrYrrruir.z5SentencePieceNormalizer.Normalize..rZr[rr\rr]br^z!SentencePieceNormalizer.NormalizecCrcr%rhrArrrrimrfz$SentencePieceNormalizer.__getstate__cCrjr%rkrlrrrrmqrnz$SentencePieceNormalizer.__setstate__)NNNNFFFr%)r r r/rrrrrBrZdelete_SentencePieceNormalizerr[rrrrrrrrrr]rirmrrrrrs0 & rcCrDr%)r SetDataDir)data_dirrrrryrFr)r)rcCshi}|jD]\}}td|r"tdd|dd}|||<q|D] \}}t|||q'dS)z1Added snake_cased method from CammelCased method.z^[A-Z]+z(?._funccs*t|turfdd|DS|S)Ncsg|]}|qSrrr+)rrrrrusz5_batchnize.._batched_func..rZr)rrAr _batched_funcs  z!_batchnize.._batched_func)rr)rrrr)rrr _batchnizes   rrB)rrrrrrr) __version__Z sentencepiece package_datac@s&eZdZdddZddZddZdS) rNcCs$||_|jdurtj|_dSdSr%)rrstderrfilenoorig_stream_fileno)rrrrrrBs z_LogStream.__init__cCs4|jdurt|j|_t|j|jdSdSr%)rosduprorig_stream_dupdup2rrArrr __enter__s z_LogStream.__enter__cCsD|jdur t|jt|j|jt|j|jdSdSr%)rrcloserrr)rrr tracebackrrr__exit__s   z_LogStream.__exit__r%)r r r/rBrrrrrrrs  r)>rrZ_swig_python_version_info __package__r rrbuiltinsr ImportErrorrr!r$r-rr.objectr2Z>ImmutableSentencePieceText_ImmutableSentencePiece_swigregisterr\Z'ImmutableSentencePieceText_swigregisterrZ,ImmutableNBestSentencePieceText_swigregisterrZ#SentencePieceProcessor_swigregisterrprrrsZ!SentencePieceTrainer_swigregisterrZ$SentencePieceNormalizer_swigregisterrrrrimportlib.resources importlibiorrrrrBrrrrrZTokenizerHZ DetokenizemZset_random_generator_seedZset_min_log_level_versionrpathrrX resourcesfilesrrrrrsv        7 H 8 S O [