o 0 i'&@sddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z m Z dejvrOeejjjjdrOejdeeejjjddlZedZGd d d e Zd$ddZd%ddZd&dd Zd'd!d"Zed#kr}edSdS)() annotationsN)Path)tqdm)AnySequence NamedTupleZ NO_LOCAL_GGUFzgguf-pyzgguf-new-metadatac@s6eZdZUded<ded<dZded<dZd ed <dS) MetadataDetailszgguf.GGUFValueTypetypervaluestr descriptionNzgguf.GGUFValueType | Nonesub_type)__name__ __module__ __qualname____annotations__r rrrj/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/gguf/scripts/gguf_new_metadata.pyrs  rreadergguf.GGUFReaderkeyr returnrcCs||}|r |SdSN) get_fieldcontents)rrfieldrrrget_field_datas r token_list Sequence[int]tokencs6fddt|D}t|dkrtdd|S)Ncsg|] \}}|kr|qSrr).0indexr r rr %szfind_token..rzUnable to find "z" in token list!) enumeratelen LookupError)rr Z token_idsrr#r find_token$s r(writergguf.GGUFWriter new_metadatadict[str, MetadataDetails]remove_metadata Sequence[str]Nonec Csv|jD]}|jtjjjks|jdrt d|jq|jtjj j r9tjj j |vr9t d|jq|j|vrHt d|jq|j d}|tj jkrX|j dnd}t|||d}||j|}|j|vrt d|jd |jd |jd |j||j=n|jdurt d |j|jdur|j|j|j|j|jdur|n|jdqtjj j |vrt d ||tjj j j|tjj j =|D]\} }t d| d |jd |j|| |j|jqd} |jD]} | | j7} || j| jj| jj| jj| j qt!d| ddd} |"|#|$|jD]} |%| j| &| jq%|'dS)NzGGUF.z Suppressing z Skipping z Removing r)rz Modifying z: "z" -> "z" zCopying zAdding chat template(s)zAdding ZWritingbyteT)ZdesctotalunitZ unit_scale)(fieldsvaluesnameggufKeysGeneral ARCHITECTURE startswithloggerdebug Tokenizer CHAT_TEMPLATEtypes GGUFValueTypeZARRAYrrgetr r Z add_key_valuer rZadd_chat_templateitemsZtensorsZn_bytesZadd_tensor_infodatashapeZdtypenbytesZ tensor_typerZwrite_header_to_fileZwrite_kv_data_to_fileZwrite_ti_data_to_fileZwrite_tensor_dataupdateclose) rr)r+r-rval_typerold_valvalr total_bytesZtensorbarrrrcopy_with_new_metadata-sR    *   (     $   rNc Cs0ddtjjjD}tdd|D}tjdd}|jdt dd|jd t d d|jd t d d d|jdt ddd|jdt ddd|jdt ddd|jdt ddd|jddt ddd|jd dt d!d"d# |d$fd%|jd&dt d'd"d# |d(fd%|jd)d*d+d,|jd-d*d.d,| t tjd"krdnd/g}tj|jrtjntjd0i}|jpg}|jrttjj|j|tjjj<|jrttjj|j|tjjj<|jrttjj|jd1rt !|jn|j|tjjj"<|j#r)t$|j#d2!}t %|}|&d3}|rttjj||tjjj"<Wdn 1s$wY|j'r:ttjj|j'|tjjj(<|rht)*d4t)*d5t)*d6|j+sht)*d7t,d8} | d9krht)-d:t.d;t)-d<|j,t/|j,d2} t0| tjjj1} t0| tjjj2pg} |j3pgD]M\} }| |vrt)*d=| d>qt4| |}ttjj5|d;d?|||| <t |d@krt)*dA|dB|d;dCt)*dD dEd|Dq|j6pgD]K\} }| |vrt)*d=| d>q|7st8dF|dGt9|}|d;kr)|t | kr)ttjj5|d?| |||| <qt8dH|dIt:j;<|j=rd|j+sdt)*d4t)*dJ|j=dKt)*d7t,d8} | d9krdt)-d:t.d;t)-dL|j=tj>|j=| | j?dM}t0| tjjj@}|durt)AdN|||_BtC| |||dS)ONcss(|]}|dsttjj|VqdS)_N)r;getattrr7r8r>r!nrrr hs&zmain..css:|]}|dr|dddtd |fVqdS)Z _token_id.r0N)endswithsplitr&rQrrrrSis8z,Make a copy of a GGUF file with new metadata)r inputz GGUF format model input filename)r helpoutputz!GGUF format model output filenamez--general-namezThe models general.namez"name")r rXmetavarz--general-descriptionzThe models general.descriptionz"Description ..."z--chat-templatez:Chat template string (or JSON string containing templates)z"{% ... %} ..."z--chat-template-configz'Config file containing chat template(s)ztokenizer_config.jsonz--pre-tokenizerzThe models tokenizer.ggml.prez"pre tokenizer"z--remove-metadataappendz/Remove metadata (by key name) from output modelz general.url)actionr rXrZz--special-tokenzSpecial token by valuez | z "")r\r rXnargsrZz--special-token-by-idzSpecial token by id0z--force store_truez$Bypass warnings without confirmation)r\rXz --verbosezIncrease output verbosityz--help)level[r chat_templatez&*** Warning *** Warning *** Warning **z=* Most metadata is required for a fully functional GGUF file,z@* removing crucial metadata may result in a corrupt output file!z<* Enter exactly YES if you are positive you want to proceed:zYES, I am sure> YESz(You didn't enter YES. Okay then, see ya!rz * Loading: zUnknown special token "z", ignoring...z= z Multiple "z" tokens found, choosing ID z0, use --special-token-by-id if you want another:z, css|]}t|VqdSr)r )r!irrrrSsz Token ID "z" is not a valid ID!z Token ID z is not within token list!z* The "z3" GGUF file already exists, it will be overwritten!z * Writing: )arch endianesszSetting custom alignment: )Dr7r8r>__dict__keysdictargparseArgumentParser add_argumentrr join parse_argsr&sysargvlogging basicConfigverboseDEBUGINFOr-Z general_namerrASTRINGr9NAMEZgeneral_descriptionZ DESCRIPTIONrdr;jsonloadsr?Zchat_template_configopenloadrBZ pre_tokenizerZPREr<warningforcerWinfoexitZ GGUFReaderrr:LISTZ special_tokenr(ZUINT32Zspecial_token_by_id isdecimalr'intospathisfilerYZ GGUFWriterriZ ALIGNMENTr=Zdata_alignmentrN)Ztokenizer_metadataZ token_namesparserargsr+r-fpconfigtemplateresponserrhrr6r idsZ id_stringZid_intr)Z alignmentrrrmaings $$ 2           "  &      r__main__)rrrr rr)rrr r rr) rrr)r*r+r,r-r.rr/)rr/) __future__rrtrmrrrr{pathlibrrtypingrrrenviron__file__parentexistsrinsertr r7 getLoggerr<rrr(rNrrrrrrs*   "    :j