o W+ i@sLddlmZmZddlZddlZddlmZddlm Z Gddde Z dS))AnyDictN)ModeKeys)OfaBasePreprocessorcseZdZdZejffdd Zdeee fdeee ffddZ deee fdeee ffdd Z deee fdeee ffd d Z Z S) OfaSudokuPreprocessorz+ OFA preprocessor for sudoku tasks c sLtt|j|||g|Ri||jjdd|_|jdd|_|jdd|_|jrg|_ g|_ t dD]-}t dD]&}|j |d|j |d|d krW|d ksc|j d |j d q=q7t |j |_ t |j |_ t ||j}t |j |g} t |j |g} t |j| |jg|_ t |j| |jg|_ d Sd S) zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) promptz solve the sudoku . seg_embeddingFmax_struct_length rrN)superr__init__cfgmodelgetinstruction_textr r input_puzzle_rowinput_puzzle_colrangeappendtorchZtensorZ zeros_like tokenize_textcatbos_itemZeos_item) selfrZ model_dirmodeargskwargsidxZjdxZ instruct_segrr __class__o/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/modelscope/preprocessors/ofa/sudoku.pyrsH          zOfaSudokuPreprocessor.__init__datareturncCs |jtjkr ||S||S)N)rrZTRAIN_build_train_sample_build_infer_sample)rr%r#r#r$__call__:s   zOfaSudokuPreprocessor.__call__cCsl||}|d}|}d|d|j}|j|dd|d<t|j |dddg|d<|S) z build sample for training tasks. step 1. execute the `_build_infer_sample` function to get a batch sample for inference. step 2. process the label data for training. label NF)Zadd_bostargetZprev_output_tokens) r(lowerstripsplitjoinZmax_tgt_lengthrrrr)rr%sampler,Ztarget_token_listr#r#r$r'@s z)OfaSudokuPreprocessor._build_train_samplecCsd|jvr d|vs Jd||jd}d|d|j}|||j}|d|j|j}d|d}|j rH|j |d<|j |d<d |jvr`|jd |vr`d ||jd |d <|S) a- build sample for inference tasks. step 1. Get the input random masked sudoku text input, which shold be generated like below pseudo code. >>> sudo = np.random.randint(1, 9, size=(9, 9)) # a pseudo sudoku >>> sudo_text = " | ".join(" : ".join(str(c) for c in row) \ >>> for row in sudo) step 2. Limit the length, tokenize the input text and add the bos token to the front of the input as source input. step 3. Add a pseodo ids for every input. textz;there must be `text` column in task key map and source datar+Ng)idsourceZseg_row_tokensZseg_col_tokensZsolutionz {}r*) Z column_mapr1r.r/r0r rrZmax_src_lengthr rrformat)rr%r3Zsrc_itemr2r#r#r$r(Qs$     z)OfaSudokuPreprocessor._build_infer_sample)__name__ __module__ __qualname____doc__rZ INFERENCErrstrrr)r'r( __classcell__r#r#r!r$r s"*"*r) typingrrnumpynprZmodelscope.utils.constantrbaserrr#r#r#r$s