o 1 i@sddlZddlmZddlmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZededeefd d Zeded edefd d ZGdddZGdddeZGdddeZGdddZdeefddZdS)N)deque)AnyCallableDequeDictListOptionalUnion)Dataset) AggregateFnV2) DeveloperAPIvaluereturncCs |dS)z*Tokenize a string using a split on spaces. )split)r rh/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/data/preprocessors/utils.pysimple_split_tokenizer s r num_featurescCs,t|}t|}t|d}||S)z6Deterministically hash a value into the integer space.)strencodehashlibsha1int hexdigest)r rZ encoded_valueZ hashed_valueZhashed_value_intrrr simple_hashs  rc @sBeZdZdZddddeeefdedeegeffdd Zd S) BaseStatSpeczEEncapsulates a statistical computation with optional post-processing.cC|SNrxrrr zBaseStatSpec.post_process_fnstat_fnr% post_key_fncCs||_||_||_dSrr&r%r')selfr&r%r'rrr__init__s zBaseStatSpec.__init__N) __name__ __module__ __qualname____doc__r r rrr*rrrrrs  rc s^eZdZdZdddddeeeegeffdedeegefd eeffd d Z Z S) AggregateStatSpecz5Represents an AggregateFnV2 spec for a single column.cCrrrr rrrr"/r#zAggregateStatSpec.N)r%column aggregator_fnr%r'r0cstj|||d||_dSNr()superr*r0)r)r1r%r'r0 __class__rrr*+s  zAggregateStatSpec.__init__) r+r,r-r.r r rrrr* __classcell__rrr4rr/(s r/c s`eZdZdZddddedeeegefdeeegefded eef fd d ZZ S) CallableStatSpeczPRepresents a user-defined stat function that operates outside Dataset.aggregate.cCrrrr rrrr"Dr#zCallableStatSpec.r$r& stat_key_fnr'r%columnscs"tj|||d||_||_dSr2)r3r*r9r8)r)r&r8r'r%r9r4rrr*>s   zCallableStatSpec.__init__) r+r,r-r.rrrrr*r6rrr4rr7;sr7c @seZdZdZddZddZdddd d eegefd ed e eegefd e eddf ddZ dddd dege fd edeegefd e eegefd e eddf ddZ dedeee ffddZde efddZde efddZde efddZdd ZdS)!StatComputationPlana\ Encapsulates a set of aggregators (AggregateFnV2) and legacy stat functions to compute statistics over a Ray dataset. Supports two types of aggregations: 1. AggregateFnV2-based aggregators, which are batch-executed using `Dataset.aggregate(...)`. 2. Callable-based stat functions, executed sequentially (legacy use case). cCs t|_dSr)r _aggregatorsr)rrrr*Xs zStatComputationPlan.__init__cCs|jdSr)r;clearr<rrrreset[szStatComputationPlan.resetcCrrrr rrrr"br#zStatComputationPlan.N)r%r'r1r%r'r9rc Cs.|D]}||}|jt||||dqdS)a Registers an AggregateFnV2 factory for one or more columns. Args: aggregator_fn: A callable (typically a lambda or class) that accepts a column name and returns an instance of AggregateFnV2. post_process_fn: Function to post-process the aggregated result. post_key_fn: Optional key generator to use to save aggregation results after post-processing. columns: List of column names to aggregate. )r1r%r'r0N)r;appendr/)r)r1r%r'r9r0Z agg_instancerrradd_aggregator^sz"StatComputationPlan.add_aggregatorcCrrrr rrrr"~r#r&r8c Cs"|jt|||||p |ddS)a Registers a custom stat function to be run sequentially. This supports legacy use cases where arbitrary callables are needed and cannot be run via Dataset.aggregate(). :param post_key_fn: :param stat_fn: A zero-argument callable that returns the stat. :param post_process_fn: Function to apply to the result. :param columns: :param stat_key_fn: )r&r%r9r8r'N)r;r?r7)r)r&r%r8r'r9rrradd_callable_statzsz%StatComputationPlan.add_callable_statdatasetc Csi}|}|r.|j|}|D]}|jj}|jdur"||jn|}|||||<q|D]!}||j }|j D]} | | }|| }|||||<q=q2|S)aq Executes all registered aggregators and stat functions. AggregateFnV2-based aggregators are batched and executed via Dataset.aggregate(). Callable-based stat functions are run sequentially. Args: dataset: The Ray Dataset to compute statistics on. Returns: A dictionary of computed statistics. N) _get_aggregate_fn_listZ aggregate_get_aggregate_specsr&namer'r0r%_get_custom_stat_fn_specsr8r9) r)rBstatsZ aggregatorsZ raw_resultspecZstat_keyZpost_keyresultcolrrrcomputes&          zStatComputationPlan.computecCdd|jDS)NcSsg|] }t|tr|jqSr) isinstancer/r&.0rHrrr s z>StatComputationPlan._get_aggregate_fn_list..r;r<rrrrCsz*StatComputationPlan._get_aggregate_fn_listcCrL)NcSg|] }t|tr|qSr)rMr/rNrrrrP  z.rQr<rrrrDz(StatComputationPlan._get_aggregate_specscCrL)NcSrRr)rMr7rNrrrrPrSzAStatComputationPlan._get_custom_stat_fn_specs..rQr<rrrrFrTz-StatComputationPlan._get_custom_stat_fn_specscCs t|S)z4 Iterates over all AggregatorSpecs. )iterrDr<rrr__iter__s zStatComputationPlan.__iter__)r+r,r-r.r*r>rrr rrr@rrAr rrKrCr/rDr7rFrVrrrrr:NsJ       % r: callbackscsfdd}|S)z Wraps a base post-processing function with a sequence of callback functions. Useful when multiple post-processing steps need to be applied in order. cs|}D]}||}q|Srr)rI processedcbbase_fnrWrrwrappers z$make_post_processor..wrapperr)r[rWr\rrZrmake_post_processorsr])r collectionsrtypingrrrrrrr Zray.datar Zray.data.aggregater Zray.util.annotationsr rrobjectrrrr/r7r:r]rrrrs  $