o
rq i
@ s U d dl Z d dlZd dlmZ d dlmZmZmZmZm Z m
Z
m Z d dl Z d dl
m mZ d dlmZ d dlmZ d dlmZ d dlmZmZ dd lmZ dd
l m Z dd l m Z m Z m!Z!m"Z"m#Z#m$Z$ dd l m%Z%m&Z&m'Z'm(Z(m)Z) e Z*d
Z+i Z,ee-ee-ef f e.d< d&de-dede e- de e/ ded ddf ddZ0dee/ de e/ee/ f dee/ fd d Z1 d'd e)d e-de
e-e%ee- f fd!d"Z2G d#d$ d$eZ3G d%d dZ4dS )( N)deepcopy)AnyDictListLiteralOptionalTupleUnion)
get_logger)Module) pad_sequence)PreTrainedTokenizerBaseStoppingCriteria )loss_scale_map)get_tools_prompt)
load_batch
load_image
rescale_image fetch_one to_device
decode_base64)HistoryPrompt StopWordsContextMessagesz You are a helpful assistant.TEMPLATE_MAPPINGdelete
template_type tokenizerdefault_system
max_lengthtruncation_strategyr Ztruncation_leftreturnTemplatec K s0 t | }t|d }|j||||fi | |S )Ntemplate)r r
init_template)r r r! r" r# kwargsZ
template_infor' r* n/home/app/PaddleOCR-VL/.venv_paddleocr/lib/python3.10/site-packages/modelscope/preprocessors/templates/base.py get_template s r,
token_listsub_token_listc C st t |tr|g}g }d}z# | |d |d }t|dks*|| ||t| kr/|| q ty9 Y |S w )z,Find the index of a token in the token_list.Tr r )
isinstanceintindexlenappend
ValueError)r- r. residxr* r* r+ _findall' s
$
r8 dHe
d2e?de
fddZ@dS )r& a A template class for all supported models.
Args:
prefix: Prefix tokens before the first turn's prompt
prompt: A list of elements whose types are str and list of integers. The input query part of every turn.
chat_sep: The chat separators between every turn.
suffix: The end tokens after the chat finished.
default_system: A default system instruction.
system_prefix: The prefix if the `system` is not empty.
auto_add_bos: By default, the bos_token is not added. The auto_add_bos option will determine
whether to add it based on `tokenizer.encode('')`.
tools_prompt: The tools prompt name
tool_prompt: The tool prompt, usually useful when there is a tool role
padding_side: The padding side
infer_media_type: The media type supported by the multi-modals
Examples:
system
You are a helpful assistant!
Who are you?
assistant:I am a robot
Who are you?
assistant:I am a robot # noqa
----------system------------ ---query---- --response- -----chatsep----- ---query--- --response- ----suffix-----
----------------------------system_prefix---------------------------- ---------------------------- prompt ------------------------------------- ---------------------------- prompt -------------------------------------
) )imagesvideosaudiosobjects norm_1000r^ TFNreact_enright
interleaveprefixpromptchat_sepsuffixr!
system_prefix auto_add_bos tools_prompt tool_prompt padding_side)leftri infer_media_type)rj Zdialogueroundr% c
C s |||||fD ]
} | d u st | tsJ q|dkrd }| |r/|d u s(J d|}| |}|| _|| _| jd u rKtdd |D sK|d u sKJ d|| _|| _| jd u| _ || _
|| _ d| _ || _
d| _|| _| d uro| n| j| _|
| _| | _d S )N z'The prefix already contains {{SYSTEM}}.c S g | ]}d |v qS
{{SYSTEM}}r* .0contextr* r* r+
z%Template.__init__..z'The template does not support `system`.TF)r0 list _has_system_replace_systemrk ro anyrl rm support_multi_roundrn r! use_default_systemrp _is_initrq rr rs ru )
rN rk rl rm rn r! ro rp rq rr rs ru xr* r* r+ rO s2
zTemplate.__init__c C s dd | D S )zReplace system with the c S s g | ] }d |v r| d dqS )rz rw )replacer| pr* r* r+ r~ s z,Template._replace_system..r* rk r* r* r+ r s zTemplate._replace_systemc C s t dd | D S )Nc S rx ry r* r r* r* r+ r~ r z(Template._has_system..)r r r* r* r+ r s zTemplate._has_systemr valuec C sb |du rdS g }|D ]$}t |tr)g }|D ]}t |tr!t| |}|| q|}|| q
|S )zRTurn `eos_token_id` to token id
e.g. [['eos_token_id']] -> [[2]]
N)r0 r rS getattrr4 )r r Z res_valuevZres_vZsub_vr* r* r+ token_attr_to_id s
zTemplate.token_attr_to_idr defaultr/ r" r# r$
loss_scaler c
K s | j du s J dd| _ || _t|dd| _|dkr d| _n|dur5| jdus2J dt| dd || _|| _|| _t|t rHt
|d| _ n|| _ || _
d D ]}t| |} | || } t| || qPdS )
a Init template by a tokenizer
Args:
tokenizer: The tokenizer to tokenize the sentence
default_system: The default system to use if the dataset does not provide one
max_length: Max length of the sequence
truncation_strategy: The truncation strategy
loss_scale: The loss scale function to use
rescale_image: Rescale image to reduce memory usage, default `-1` means no limitation
Fz"The template has been initialized.T
is_multimodalNrw 7The template does not support `system`, template_type: r )rk rl rm rn ro )r r r r r! ro r" r# r0 rS r getr r r setattr)
rN r r! r" r# r r r) keyr r* r* r+ r( s,
zTemplate.init_templatemodeldatac C i S )a This method will be called after data_collator and before the forward
Args:
data: The `_data` field from the example batch, this field should be packed manually
Returns:
Any extra fields need to be passed into the model.forward
r* )rN r r r* r* r+ post_encode s zTemplate.post_encodeexamplec C s dS )zCheck example validNr* )rN r r* r* r+
check_example zTemplate.check_examplec
s |d }dD ]\}}| |rdd |D t }|d dks"J fddt|d D }| jdkrtt|d ||| D ]A\}}} tt||d d }
| rt|
d
kscJ d |
d |d |
dkrs||d d |d d < q@|
dksJ d
|d q@dd || D ||< qtt|ddd |D }
dd || D ||< t|| } | |
} | dksJ d| d|
|| |d d d |d d d < qdS )a: Add default tags to example, this is for the multi-modal datasets
1. For the round infer_media_type, this method will check the tag equals with the chat round
2. Else, this method will try to add tags to the head of the messages
Args:
example: The input example
r: ))rd r_ )rc r^ )re r` c S g | ]
}|d dkr|qS r= r? r* r| messager* r* r+ r~ z-Template.add_default_tags.. r c s g | ]
} ||d qS )r r* r| rG Z _messagesr* r+ r~ r rv r< r zMThe model includes at most one media per round. However, this round contains z media_tags. query: zMissing media. query: c S g | ]}|r|qS r* r* r| rH r* r* r+ r~ r
c S s g | ]}|d d qS )r r< r* )r| hr* r* r+ r~ c S r r* r* r r* r* r+ r~ r zNumber of media: z, number of media_tags: N)r r3 rangeru ziprB rC join)
rN r r: media_keyZ media_tagZn_roundhistoryrG r rH Znum_media_tagsZ num_mediaZ num_new_tagsr* r r+ add_default_tags s>
"
$zTemplate.add_default_tagsc C s | j dv r't|dd\|d< }|dr |r td|dp$g | |d< | j dv rVtddgdd gd
d gD ] \}}}t|d||\|d< }||pRg | ||< q8d S d S )
zzReplace the tag
Args:
example: The input example
> TNr: r^ rc z3Do not mix use the tag.re rd r` r_ z(.+?) z(.+?) N)r rI r r5 r )rN r rF ktagrE Z medias_pathr* r* r+ replace_media_tags s"
zTemplate.replace_media_tagsc sX dddd}| D ]}||r!t|| ttfs!|| g||< q
| | |drct|d trct |d |d< g }|d D ]}t|trY|d |d ddd }|
| qG||d< |dpig }|r|dst j ryt |t
}|dr j|d | jd
j r jd kr fd d
|D } j st|dd }||d< dS dS )ap Preprocess multi-modal media resources in one example
1. Wrap all values in media keys to list
2. Replace .)rc )valuesr r0 tupler r r rS jsonloadsr4 load_mediasr r normalize_bboxgrounding_typer )rN r Zmultimodal_keysr rf objectrc r* r r+ _preprocess_media6 sD
zTemplate._preprocess_mediac C s* | }t| dd }|dpg }| jstd|d }dd |D }dd |D }|r2|d d nd
}|s=| jr<| j}n | jd usIJ d | |rct|t rUt
|}|d u r[d
}|t || j
7 }|rs|smd d d
g}||d d < t|dkr| jsJ d| || |d< | | | | |S )Nr toolsz[Template is not initialized, please use the `get_template` function to obtain the template.r: c S g | ]
}|d dkr|qS r r* r r* r* r+ r~ z r z'Template.preprocess..c S r r r* r r* r* r+ r~ { r r r< rw r r? )r= r< r z?The template does not support multi-round chat, template_type: )rA r r r r5 r r! ro r0 rS r r r rq r3 r r r )rN r r r r: Z system_roundr? r* r* r+
preprocessn sH
zTemplate.preprocess streaming is_trainingc K sz | |}| j|fi |}|d }|s7d|v r7|d}|dus#J |d}t||j}|| || |s;|S |S )a The entrance method of Template!
Args:
example: The input example
streaming: If is streaming mode
is_training: Use template in training
**kwargs:
model: The model instance, use only in `is_training=False`
Returns:
if not streaming mode, returns tuple of (example, tokenizer_kwargs), else return example only
r _datar N)r _encoder popr Zdeviceupdater ) rN r r r r) r6 inputsr r r* r* r+ encode s
zTemplate.encodec s^ d }t fddtjD }| j|| j| j| d\}}|ddu r+|dd ||fS )z return: inputs, tokenizer_kwargsr: c s g | ]} |qS r* r )r| r r r* r+ r~ z$Template._encode..)rp is_multi_modalr labelsNr )r r& special_keys_concat_and_tokenizer# rp r r )rN r r) r: r r rL r* r r+ r s
zTemplate._encode context_listres_context_listloss_scale_listr? queryresponseround0 compute_lossc C s d} |durt |d } t |}|D ]a}
t|
t rbd|
kr@|dus#J |r.| ||\} } n|gdg} } || || qg d}
|||| g}t|
|D ]\}}|dura||
v ra|
||}
qOt|
dkriq||
|d qdS )z+Concat context list and replace placeholderNr {{RESPONSE}} )rz z {{QUERY}}z
{{ROUND0}}z
{{ROUND1}}r )rS r0 r extendr r r3 r4 )rN r r r r? r r r r Zround1r} Z content_partZ weight_partZ old_str_listZ new_str_listZold_strZnew_strr* r* r+ _concat_context_list s2
z Template._concat_context_listc K s | dd}|r| ||\}}| j||fi |\}}g }g }g }d}tt||D ]C\} \}
} t|
tr@| |kr@||
q+t|dkrW|d | || |
t|
trb||
n
||
|| | }q+t|dkr|d | || ||fS )z4Merge anything in the context to simplify the inputsr Fr r rw ) r split_special_tokens pre_tokenizer@ r r0 rS r4 r3 r clear) rN r r r) r r6 res_loss_scaletempZtemp_loss_scalerG r} r r* r* r+ _simplify_context_list s0
z Template._simplify_context_listc C s ddl m} g }g }t| |D ]B\}}g }tt|trG||tjD ]
}||d |d g q"dd |D }|| ||gt | q|
| |
| q||fS )z`Split special tokens, for example ``, ``, this will help the replace_tag operationr )split_str_parts_byr r< c S r r* r* )r| cr* r* r+ r~ r z1Template.split_special_tokens..) utilsr r r0 r rS r& special_tokensr r3 r4 ) r r r r6 Zloss_scale_resr} r Zcontextsdr* r* r+ r s
z Template.split_special_tokensc K s | j |fddd|d S )NF)Zreturn_attention_maskZadd_special_tokensrP )r )rN r} rL r* r* r+ _tokenize s zTemplate._tokenize
media_typer r r r2 c C s. |dkr| j S |dkrdgS |dkrdgS dS )a Override this function to do your own replace operation.
This method is used to replace standard tags like `` to some tokens that the model needs.
Args:
media_type: The modal.
index: The index of the medias, for example 0 represents the first elements in `images`
example: The input example
Returns:
The content or input_ids after replacement.
r r r_ r r` N)image_placeholder)rN r r2 r r* r* r+ replace_tag s zTemplate.replace_tagc C s& | d}|r|| }|d gS dgS )aY Replace objects referenced by the bbox to contents or input_ids. This is useful in the grounding task.
Override this function to do your own replace operation.
Args:
index: The index in the `objects` key
example: The input example
Returns:
The contents or input_ids replaced
rf r rb r )rN r2 r rf object_r* r* r+ replace_object1 s
zTemplate.replace_objectc C s | d}|r_|| }t|d d tr?d}|d D ]}|d|d d|d d|d d|d
d 7 }q|d d
}|gS d|d d d|d d d|d d d|d d
d gS dgS )aW Replace bbox pointing to the objects to contents or input_ids. This is useful in the grounding task.
Override this function to do your own replace operation.
Args:
index: The index in the `objects` key
example: The input example
Returns:
The contents or input_ids replaced
rf r r rw z[(,r z),(r z)],Nr/ z)]ra )r r0 r )rN r2 r rf r Z all_objectsZ
sub_objectr* r* r+ replace_boxC s
4 @zTemplate.replace_boxrf rc r )r rg norm_1c
s |r|sdS |D ]}|d }|d }|d }|| }|dkrg dkr#q|j |j} }
t|d trPg } |D ]} | fddt| | |
| |
gD q5| |d< n fd dt|| |
| |
gD |d< |d< q|d
kr d
krpq d kr~d d |D |d< n dkr|j |j} }
d
d t|| |
| |
gD |d< |d< q|d krԈ d krq d
krdd |D |d< n dkr|j |j} }
dd t|| |
| |
gD |d< |d< qdS )a] Normalize bbox to needed.
to_type support real/norm_1000/norm_1, which literally means the coordinates in real, or normalized by 1000,
or normalized by 1.
Args:
objects: The objects containing the bbox
images: The images list
to_type: The coordinate type needed by the model.
Nr r r r r c 0 g | ]\}} d krt || d n|| qS rg r1 r| coorddimr r* r+ r~ w z+Template.normalize_bbox..c r r r r r r* r+ r~ } r rg r c S s g | ]}|d qS g 8@r* r| r r* r* r+ r~ r c S s g | ] \}}t |d | qS r r r r* r* r+ r~ s c S s g | ]}t |d qS )r r r r* r* r+ r~ r c S s g | ]
\}}t || qS r* r r r* r* r+ r~ r )widthheightr0 r r4 r )
clsrf rc r r r r r7 r r r Zbboxes_boxr* r r+ r \ sX
zTemplate.normalize_bboxc K s | d}g }g }dD ] }d|| d< q t||D ]l\}} dD ]#}|d| dkrC| ||| d |}
|| d d7 < n6q |dkr]| | d d|}
| d dd |d < n |d
krv| | d d|}
| d dd |d < n|g}
||
7 }|| gt|
7 }q||fS ) aA This method happens before tokenization, replace standard tags to the contents or input_ids needed by
the model.
Args:
context_list: The content list
loss_scale_list: The loss scale list
Returns:
The context_list and loss_scale_list after replacement.
r r r _index<>r rb Z object_indexra Z box_index)r r r r r r3 ) rN r r r) r r6 r r r} r Zc_listr* r* r+ r s,
zTemplate.pre_tokenizec C s g }g }g }i }|du rdgt | }tt||D ]D\}\}} t|tr:| |}
| ||
| j|fi |
} n|} || 7 }|| dkrK|| 7 }n |dgt | 7 }|| gt | q||||fS )z+return: input_ids, labels, tokenizer_kwargsNr ) r3 r@ r r0 rS _get_tokenizer_kwargs_concat_tokenizer_kwargsr r ) rN r r rP r r rL rG r} Z loss_weightcurr_tokenizer_kwargsr- r* r* r+ _encode_context_list s$
z Template._encode_context_listr suffix_tokens_idc C s t |}d}tdt | D ]4}| |d dkr | | dkr |}|dkrA| |d dkrA| | dkrA|| }||krA|| ||| < q
d S )Nr r r )r3 r )r r Z
suffix_lenstartrG lengthr* r* r+ use_dynamic_eos s $zTemplate.use_dynamic_eosr: c s dd D }dd D t |dkr|d d }nd}t dks%J t dkrK d d kr>d d g}d d g}n5 d dg} d dg}n(t d
dksUJ fd dtt d
D } fd dtt d
D }g }g } |r| jj}
t|
tr|
| jd
v r||
g | d | j } |du rdd | D } |du st
dd | D r| j } n| j } | j
| || |d tt||D ]v\}
\\}}\}}|dkr| j n| }g }d}|
t |d k rdd |D }|d ||
d d r| j}n|dur|d | j}d}|s|r?| j
||| ||||
| jp)|d ||7 }| |r7dgndgt | 7 } qi }| jrt |t|d d du }t |}tddgt|| |td|| gD ]9\}}| j|| | | fi |\}}| ||\} } } } | | || d < || d < | jr| || d < qk|d |d } |d! |d" } |d d du rt |d" dksJ d|d"< n'| j|| fi |\}} | || \} } } } | dur| | | | jd |d d du r d} | jdurH|d#kr&t | | jkr&t d$t | d%| j d& i i fS | | j d } | dur;| | j d } | durH| | j d } | |d'< | |d(< | jrX| |d)< || fS )*z2
return: inputs, tokenizer_kwargs
c S r r r* r r* r* r+ r~ r z1Template._concat_and_tokenize..c S r r r* r r* r* r+ r~ r r r< Nr r= r r c ( g | ]} | d |d d gqS )r< r r* r r: r* r+ r~ ( c r )r= r r* r r r* r+ r~ r rw r c S g | ]}d |vr|qS ry r* r{ r* r* r+ r~
r c S rx ry r* r{ r* r* r+ r~ r )r? r> Fc S r ry r* r{ r* r* r+ r~ r r T)r r r? r r g ?r/ Zanswerrl Z
_input_idsZ_labelsZ _loss_scaleZprompt_input_idsZanswer_input_idsZ
prompt_labelsZ
answer_labelsr zCurrent length of row(z ) is larger than the max_length(z ), deleted.rP r r ) r3 r r bos_token_idr0 r1 r r4 rl rA r rk ro r r@ r rr rm rn compute_per_round_lossoutput_prompt_answerr] slicer r r r r" loggerwarn) rN r: r# rp r) r? r Z
history_rolesr r r rl rk rG qrZqrrrr Zextra_context_listZ is_suffixr Z
answer_lenZ total_lenr Z_sliceZ_res_context_listZ_loss_scale_listrP r r rL r* r r+ r s
"
z Template._concat_and_tokenizer} c C r )z return: curr_tokenizer_kwargsr* )rN r} r* r* r+ r
[ r z Template._get_tokenizer_kwargsrL r c C s t |dksJ d S )Nr )r3 )rN rL r r* r* r+ r _ s z!Template._concat_tokenizer_kwargsr sequences
padding_value)ri rt c
C s |dk}|r
t | d|dS tdd | D }g }| D ]'}||d }dg| d d |dg }t|t|d |} || qt |S )
zPad sequence by some side
Args:
sequences: The input sequences in tensor.
padding_value: The padding value
padding_side: The padding side
Returns:
A tensor after padding
ri T)Z batch_firstr# c S s g | ]}| d qS )r )size)r| sr* r* r+ r~ t r z)Template.pad_sequence..r r r constant)
r maxr$ r Fpadr r4 r[ stack)
r" r# rs
padding_rightmax_lenZpadded_sequencesseqZ
pad_lengthZ pad_tupleZ
padded_seqr* r* r+ r b s
zTemplate.pad_sequencebatch
padding_toc sD | j }|jdus
J | jdk}i }d|d v r2dd |D |d< fddttD |d< n d |d v rRd
d |D |d < fd dtt D |d< d D ]|d v rgfd
d|D |< qT|durd |v srJ ||d d jd }|dkrtg d|jddddgD ] \}|v rt| d |rd|fn|dfd|| d< qtg d|jdddddgD ]\}|v r| | || j|< qd|d v rdd |D |d< dd |D }t|dkr t
||d< dd |D } t| dkr t
| |d< dd |D }
t|
dkr t
|
|d< |S ) z
Args:
batch(`List[Dict[str, Any]]`): The input data in batch
padding_to(`int`, optional): Whether padding the batch to a fixed length, if none, the batch
will be padded to the `longest`
Nri
inputs_embedsr c S g | ]}|d qS r0 r* r| br* r* r+ r~ r z*Template.data_collator..c s& g | ]}t j | jd t jdqS )r Zdtype)r[ onesshapeint64r r2 r* r+ r~ s attention_maskrP c S s g | ] }t |d qS rP r[ Ztensorr3 r* r* r+ r~ c s$ g | ]}t jt | t jd qS )r5 )r[ r6 r3 r8 r r: r* r+ r~ s $ )r r position_idsc s g | ] }t | qS r* r; r3 )r r* r+ r~ r< r/ )rP r9 r r r= r r r& )rP r0 r9 r r r= r c S r1 )r r* r3 r* r* r+ r~ r c S " g | ]
}| d dur|d qS ) pixel_valuesNr r3 r* r* r+ r~ " r? c S r> ) image_sizesNr r3 r* r* r+ r~ r@ rA c S r> )pixel_values_videosNr r3 r* r* r+ r~ r@ rB ) r Z pad_token_idrs r r3 r7 r r( r) r r[ concat) rN r. r/ r r+ r6 Z padding_lenr r? rA rB r* )rP r0 r r+
data_collator s`
zTemplate.data_collator generate_idsinput_token_lenc C sF t |tjr
| }t|dkr t |d ttfr |d }| ||S )Nr r )r0 r[ r\ rT r3 r r _get_generate_ids)r rE rF r* r* r+ get_generate_ids s
zTemplate.get_generate_idsc C s | |d S Nr* )rE rF r* r* r+ rG s zTemplate._get_generate_idscpc C s d| kr
dks[n d| krdks[n d| kr dks[n d| kr+dks[n d | kr6d
ks[n d | krAd ks[n d
| krLdks[n d| krXdkr]dS dS dS dS )z6Checks whether CP is the codepoint of a CJK character.i N i i 4 iM i iߦ i i? i@ i i i i i i i TFr* )rJ r* r* r+ _is_chinese_char s BB(zTemplate._is_chinese_char print_idx is_finishedc C sX |rt |S |dst |dkr | t|d r t |}|S t|dd |}|S )Nr r r/ r )r3 endswithrK ordr' rfind)r r rL rM r* r* r+ _get_safe_print_idx s (z Template._get_safe_print_idx)rL return_deltarL first_num_spacerS rT c
C s |d u ri }| j }t|dr| }t| jd tr;|r/|r;|t| jd d | jd kr;|d t| jd }|rJ|rP|dd | j jgkrP|d d }|j|fi |}|d ur|} |d }t|t| d }
|sz|dkrz|
}|| d< |
|k rd||
| }n |
|kr||
| d }t| jd t
r|r|r|t| jd d | jd krt t|t| jd d} |d urt | |d } |d | }|d ur|d } |s| ||d |d< |d |d }|r|| d }|S |r|rJ |S )NrT r/ r rN )
r hasattrrT r0 rn r r3 Z eos_token_idrR lstriprS r' rR )
rN rE rM rL rS rL rT r r Zres_fnsZ
cur_num_spacer7 Z
old_print_idxr* r* r+ generate_ids_to_response sX
"
" z!Template.generate_ids_to_responsec C s |S rI r* )rN r r r* r* r+ post_process_generate_response s z'Template.post_process_generate_response)NNFrh Nri rj )NNr r r/ )r% N)FF)NNNNTrI )F)r ri )T)ArW rX rY rZ r r r r r r r r r rS r] r rO staticmethodr r r
r r1 r( r r r r r r r r r r r r r r floatr r r r r r r classmethodr r r r r r
r r[ r\ r rD rH rG rK rR rW dictrX r* r* r* r+ r& c st
'
)
"8<+2
$
!
"" :
(
"
v& 0 9
7)NNr )r9 )5r rB rA r typingr r r r r r r r[ Ztorch.nn.functionalnnZ
functionalr( Z
modelscoper
Ztorch.nnr Ztorch.nn.utils.rnnr Z transformersr
r r r rq r r r r r r r r r r r r r r ZDEFAULT_SYSTEMr rS __annotations__r1 r, r8 rI rJ r& r* r* r* r+ sT
$
*