o 1 i}@sddlmZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl ZddlZddlZddlmZddlmZmZddlmZddlmZerNdd lmZed d Gd d d eZed d GdddeZed d GdddeZed d GdddeZ ed d GdddeZ!dddddde e"dede#de ee"e$ff ddZ%d+d!e#d"ee gee"e$fffd#d$Z&d%ej'de"d"dfd&d'Z(d(ej)d"e#fd)d*Z*dS),)Counter)partial) TYPE_CHECKINGAnyCallableDictHashableListOptionalSetN) BatchFormat) PreprocessorPreprocessorNotFittedException)make_post_processor) PublicAPI)Datasetalpha)Z stabilityc sleZdZdZddddeededeeeffdd Zd d d e fd dZ de j fddZ ddZZS)OrdinalEncodera Encode values within columns as ordered integer values. :class:`OrdinalEncoder` encodes categorical features as integers that range from :math:`0` to :math:`n - 1`, where :math:`n` is the number of categories. If you transform a value that isn't in the fitted datset, then the value is encoded as ``float("nan")``. Columns must contain either hashable values or lists of hashable values. Also, you can't have both scalars and lists in the same column. Examples: Use :class:`OrdinalEncoder` to encode categorical features as integers. >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import OrdinalEncoder >>> df = pd.DataFrame({ ... "sex": ["male", "female", "male", "female"], ... "level": ["L4", "L5", "L3", "L4"], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OrdinalEncoder(columns=["sex", "level"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sex level 0 1 1 1 0 2 2 1 0 3 0 1 :class:`OrdinalEncoder` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = OrdinalEncoder(columns=["sex", "level"], output_columns=["sex_encoded", "level_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sex level sex_encoded level_encoded 0 male L4 1 1 1 female L5 0 2 2 male L3 1 0 3 female L4 0 1 If you transform a value not present in the original dataset, then the value is encoded as ``float("nan")``. >>> df = pd.DataFrame({"sex": ["female"], "level": ["L6"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(ds).to_pandas() # doctest: +SKIP sex level 0 0 NaN :class:`OrdinalEncoder` can also encode categories in a list. >>> df = pd.DataFrame({ ... "name": ["Shaolin Soccer", "Moana", "The Smartest Guys in the Room"], ... "genre": [ ... ["comedy", "action", "sports"], ... ["animation", "comedy", "action"], ... ["documentary"], ... ], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OrdinalEncoder(columns=["genre"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [2, 0, 4] 1 Moana [1, 2, 0] 2 The Smartest Guys in the Room [3] Args: columns: The columns to separately encode. encode_lists: If ``True``, encode list elements. If ``False``, encode whole lists (i.e., replace each list with an integer). ``True`` by default. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`OneHotEncoder` Another preprocessor that encodes categorical data. TN) encode_listsoutput_columnscolumnsrrcs(t||_||_t|||_dSN)super__init__rrr #_derive_and_validate_output_columnsr)selfrrr __class__j/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/data/preprocessors/encoder.pyris  zOrdinalEncoder.__init__datasetrreturnc0jjfddtddddjdS)Ncstjj|dS)N)r rrkey_gen)compute_unique_value_indicesrrr#r rrrzs z%OrdinalEncoder._fit..cS d|dSNzunique()rcolrrrr' cSr(Nunique_values(r*rr+rrrr'r-Zstat_fnZpost_process_fnZ stat_key_fnZ post_key_fnrstat_computation_planadd_callable_statunique_post_fnrrr rr&r_fitxs  zOrdinalEncoder._fitdfcsXt|gjRdtdtffdd dtjffdd }|j||j<|S)Nelementnamecsfdd|DS)Ncs$g|]}jdd|qS)r/r*)stats_get.0x)r9rrr s$zIOrdinalEncoder._transform_pandas..encode_list..r)r8r9rr9r encode_listsz5OrdinalEncoder._transform_pandas..encode_listscsXtrjrtjdSfdd}|Sjdjd}|S)NrAcs"t|}jdjd|Sr.)tupler:r9r;)r8key)rCrrrlist_as_categoryszZOrdinalEncoder._transform_pandas..column_ordinal_encoder..list_as_categoryr/r*)_is_series_composed_of_listsrmaprr9applyr:)rCrFs_valuesrBr)rCrcolumn_ordinal_encoders  z@OrdinalEncoder._transform_pandas..column_ordinal_encoder) _validate_dfrliststrpdSeriesrIr)rr7rLrrKr_transform_pandass z OrdinalEncoder._transform_pandascC&|jjd|jd|jd|jdS)N (columns=z, encode_lists=, output_columns=r*)r__name__rrrr@rrr__repr__ zOrdinalEncoder.__repr__)rV __module__ __qualname____doc__r rOboolr rr r6rP DataFramerRrW __classcell__rrrrrsY rc seZdZdZddddeedeeeefdeeeffddZ d d d e fd d Z de deeeffddZ dejfddZddZZS) OneHotEncodera- `One-hot encode `_ categorical data. This preprocessor transforms each specified column into a one-hot encoded vector. Each element in the vector corresponds to a unique category in the column, with a value of 1 if the category matches and 0 otherwise. If a category is infrequent (based on ``max_categories``) or not present in the fitted dataset, it is encoded as all 0s. Columns must contain hashable objects or lists of hashable objects. .. note:: Lists are treated as categories. If you want to encode individual list elements, use :class:`MultiHotEncoder`. Example: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import OneHotEncoder >>> >>> df = pd.DataFrame({"color": ["red", "green", "red", "red", "blue", "green"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OneHotEncoder(columns=["color"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color 0 [0, 0, 1] 1 [0, 1, 0] 2 [0, 0, 1] 3 [0, 0, 1] 4 [1, 0, 0] 5 [0, 1, 0] OneHotEncoder can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = OneHotEncoder(columns=["color"], output_columns=["color_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color color_encoded 0 red [0, 0, 1] 1 green [0, 1, 0] 2 red [0, 0, 1] 3 red [0, 0, 1] 4 blue [1, 0, 0] 5 green [0, 1, 0] If you one-hot encode a value that isn't in the fitted dataset, then the value is encoded with zeros. >>> df = pd.DataFrame({"color": ["yellow"]}) >>> batch = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(batch).to_pandas() # doctest: +SKIP color color_encoded 0 yellow [0, 0, 0] Likewise, if you one-hot encode an infrequent value, then the value is encoded with zeros. >>> encoder = OneHotEncoder(columns=["color"], max_categories={"color": 2}) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color 0 [1, 0] 1 [0, 1] 2 [1, 0] 3 [1, 0] 4 [0, 0] 5 [0, 1] Args: columns: The columns to separately encode. max_categories: The maximum number of features to create for each column. If a value isn't specified for a column, then a feature is created for every category in that column. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`MultiHotEncoder` If you want to encode individual list elements, use :class:`MultiHotEncoder`. :class:`OrdinalEncoder` If your categories are ordered, you may want to use :class:`OrdinalEncoder`. Nmax_categoriesrrrarc,t||_|p i|_t|||_dSrrrrrar rrrrrarrrrr   zOneHotEncoder.__init__r rr!cr")Nctjd|jdS)NFr rrr#rar$rrar%r&rrr'z$OneHotEncoder._fit..cSr(r)rr+rrrr'r-cSr(r.rr+rrrr'r-r0r1r5rr&rr6  zOneHotEncoder._fitvstatscCs2t|ttjfr t|}t|tr||dSdS)N) isinstancerNnpndarrayrDrr;)rrkrlrrrsafe_get s   zOneHotEncoder.safe_getr7c st|gjRtjjD]D\}}jd|dt}tjt||ftjd}|| fdd }|dk}t |d}d||||f<| ||<q|S) Nr/r*)dtypecs |Sr)rq)rkrrlrrr'1r-z1OneHotEncoder._transform_pandas..rmr) rMrziprr:lenroZzerosZuint8rIZto_numpyZnonzerotolist) rr7column output_columnZnum_categoriesZone_hotcodesZvalid_category_maskZnon_zero_indicesrrsrrR(s zOneHotEncoder._transform_pandascCrSNrTz, max_categories=rUr*rrVrrarr@rrrrW?rXzOneHotEncoder.__repr__)rVrYrZr[r rOr rintrr r6rrqrPr]rRrWr^rrrrr_s] r_c sxeZdZdZddddeedeeeefdeeeffddZ d d d e fd d Z de j fddZddZZS)MultiHotEncoderaMulti-hot encode categorical data. This preprocessor replaces each list of categories with an :math:`m`-length binary list, where :math:`m` is the number of unique categories in the column or the value specified in ``max_categories``. The :math:`i\\text{-th}` element of the binary list is :math:`1` if category :math:`i` is in the input list and :math:`0` otherwise. Columns must contain hashable objects or lists of hashable objects. Also, you can't have both types in the same column. .. note:: The logic is similar to scikit-learn's [MultiLabelBinarizer][1] Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MultiHotEncoder >>> >>> df = pd.DataFrame({ ... "name": ["Shaolin Soccer", "Moana", "The Smartest Guys in the Room"], ... "genre": [ ... ["comedy", "action", "sports"], ... ["animation", "comedy", "action"], ... ["documentary"], ... ], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> encoder = MultiHotEncoder(columns=["genre"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [1, 0, 1, 0, 1] 1 Moana [1, 1, 1, 0, 0] 2 The Smartest Guys in the Room [0, 0, 0, 1, 0] :class:`MultiHotEncoder` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = MultiHotEncoder(columns=["genre"], output_columns=["genre_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre genre_encoded 0 Shaolin Soccer [comedy, action, sports] [1, 0, 1, 0, 1] 1 Moana [animation, comedy, action] [1, 1, 1, 0, 0] 2 The Smartest Guys in the Room [documentary] [0, 0, 0, 1, 0] If you specify ``max_categories``, then :class:`MultiHotEncoder` creates features for only the most frequent categories. >>> encoder = MultiHotEncoder(columns=["genre"], max_categories={"genre": 3}) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [1, 1, 1] 1 Moana [1, 1, 0] 2 The Smartest Guys in the Room [0, 0, 0] >>> encoder.stats_ # doctest: +SKIP OrderedDict([('unique_values(genre)', {'comedy': 0, 'action': 1, 'sports': 2})]) Args: columns: The columns to separately encode. max_categories: The maximum number of features to create for each column. If a value isn't specified for a column, then a feature is created for every unique category in that column. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`OneHotEncoder` If you're encoding individual categories instead of lists of categories, use :class:`OneHotEncoder`. :class:`OrdinalEncoder` If your categories are ordered, you may want to use :class:`OrdinalEncoder`. [1]: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html Nr`rrarcrbrrcrdrrrrrezMultiHotEncoder.__init__r rr!cr")Ncrf)NTrgrhr%r&rrr'riz&MultiHotEncoder._fit..cSr(r)rr+rrrr'r-cSr(r.rr+rrrr'r-r0r1r5rr&rr6rjzMultiHotEncoder._fitr7cs^t|gjRdtdtffdd }tjjD]\}}||t||d||<q|S)Nr8r9csRt|tjr |}nt|ts|g}jd|d}t|fdd|DS)Nr/r*csg|]}|dqS)r)r;r<counterrrr?zJMultiHotEncoder._transform_pandas..encode_list..)rnrorprwrNr:r)r8r9rlr@rrrBs   z6MultiHotEncoder._transform_pandas..encode_listrA)rMrrNrOrurrHr)rr7rBrxryrr@rrRs  z!MultiHotEncoder._transform_pandascCs&|jjd|jd|jd|jdSr{r|r@rrrrWrXzMultiHotEncoder.__repr__)rVrYrZr[r rOr rr}rr r6rPr]rRrWr^rrrrr~GsT r~csxeZdZdZdddedeeffddZdd d efd d Zd e j fddZ dddZ d e j fddZ ddZZS) LabelEncodera Encode labels as integer targets. :class:`LabelEncoder` encodes labels as integer targets that range from :math:`0` to :math:`n - 1`, where :math:`n` is the number of unique labels. If you transform a label that isn't in the fitted datset, then the label is encoded as ``float("nan")``. Examples: >>> import pandas as pd >>> import ray >>> df = pd.DataFrame({ ... "sepal_width": [5.1, 7, 4.9, 6.2], ... "sepal_height": [3.5, 3.2, 3, 3.4], ... "species": ["setosa", "versicolor", "setosa", "virginica"] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> from ray.data.preprocessors import LabelEncoder >>> encoder = LabelEncoder(label_column="species") >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species 0 5.1 3.5 0 1 7.0 3.2 1 2 4.9 3.0 0 3 6.2 3.4 2 You can also provide the name of the output column that should hold the encoded labels if you want to use :class:`LabelEncoder` in append mode. >>> encoder = LabelEncoder(label_column="species", output_column="species_encoded") >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species species_encoded 0 5.1 3.5 setosa 0 1 7.0 3.2 versicolor 1 2 4.9 3.0 setosa 0 3 6.2 3.4 virginica 2 If you transform a label not present in the original dataset, then the new label is encoded as ``float("nan")``. >>> df = pd.DataFrame({ ... "sepal_width": [4.2], ... "sepal_height": [2.7], ... "species": ["bracteata"] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species 0 4.2 2.7 NaN Args: label_column: A column containing labels that you want to encode. output_column: The name of the column that will contain the encoded labels. If None, the output column will have the same name as the input column. .. seealso:: :class:`OrdinalEncoder` If you're encoding ordered features, use :class:`OrdinalEncoder` instead of :class:`LabelEncoder`. N)ry label_columnrycst||_|p ||_dSr)rrrry)rrryrrrrs zLabelEncoder.__init__r rr!cs2jjfddtddddjgdS)Ncstjg|dSN)r rr#)r$rr%r&rrr's z#LabelEncoder._fit..cSr(r)rr+rrrr' r-cSr(r.rr+rrrr'!r-r0)r2r3r4rr5rr&rr6s  zLabelEncoder._fitr7cs:t|jdtjffdd }|j||j<|S)NrCcsjd|jd}||Sr.)r:r9rH)rCrJr@rrcolumn_label_encoder)s z.column_label_encoder)rMrrPrQ transformry)rr7rrr@rrR&s zLabelEncoder._transform_pandasdscCsF|}|tjjtjjfvrtd|}|j|jfdt j i|S)a/Inverse transform the given dataset. Args: ds: Input Dataset that has been fitted and/or transformed. Returns: ray.data.Dataset: The inverse transformed Dataset. Raises: PreprocessorNotFittedException: if ``fit`` is not called yet. z1`fit` must be called before `inverse_transform`, batch_format) fit_statusr Z FitStatusZPARTIALLY_FITTEDZ NOT_FITTEDrZ_get_transform_config map_batches_inverse_transform_pandasr ZPANDAS)rrrkwargsrrrinverse_transform0s  zLabelEncoder.inverse_transformcs.dtjffdd }|j||j<|S)NrCcs,ddjdjdD}||S)NcSi|]\}}||qSrr)r=rEvaluerrr OszXLabelEncoder._inverse_transform_pandas..column_label_decoder..r/r*)r:ritemsrH)rCZinverse_valuesr@rrcolumn_label_decoderNs  zDLabelEncoder._inverse_transform_pandas..column_label_decoder)rPrQryrr)rr7rrr@rrMs z&LabelEncoder._inverse_transform_pandascCs|jjd|jd|jdS)Nz(label_column=z, output_column=r*)rrVrryr@rrrrWZszLabelEncoder.__repr__)rrr!r)rVrYrZr[rOr rr r6rPr]rRrrrWr^rrrrrs @  rc sxeZdZdZ  ddeedeeeej fdeeeffdd Z dd d e fd d Z d ej fddZddZZS) Categorizera^ Convert columns to ``pd.CategoricalDtype``. Use this preprocessor with frameworks that have built-in support for ``pd.CategoricalDtype`` like LightGBM. .. warning:: If you don't specify ``dtypes``, fit this preprocessor before splitting your dataset into train and test splits. This ensures categories are consistent across splits. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import Categorizer >>> >>> df = pd.DataFrame( ... { ... "sex": ["male", "female", "male", "female"], ... "level": ["L4", "L5", "L3", "L4"], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> categorizer = Categorizer(columns=["sex", "level"]) >>> categorizer.fit_transform(ds).schema().types # doctest: +SKIP [CategoricalDtype(categories=['female', 'male'], ordered=False), CategoricalDtype(categories=['L3', 'L4', 'L5'], ordered=False)] :class:`Categorizer` can also be used in append mode by providing the name of the output_columns that should hold the categorized values. >>> categorizer = Categorizer(columns=["sex", "level"], output_columns=["sex_cat", "level_cat"]) >>> categorizer.fit_transform(ds).to_pandas() # doctest: +SKIP sex level sex_cat level_cat 0 male L4 male L4 1 female L5 female L5 2 male L3 male L3 3 female L4 female L4 If you know the categories in advance, you can specify the categories with the ``dtypes`` parameter. >>> categorizer = Categorizer( ... columns=["sex", "level"], ... dtypes={"level": pd.CategoricalDtype(["L3", "L4", "L5", "L6"], ordered=True)}, ... ) >>> categorizer.fit_transform(ds).schema().types # doctest: +SKIP [CategoricalDtype(categories=['female', 'male'], ordered=False), CategoricalDtype(categories=['L3', 'L4', 'L5', 'L6'], ordered=True)] Args: columns: The columns to convert to ``pd.CategoricalDtype``. dtypes: An optional dictionary that maps columns to ``pd.CategoricalDtype`` objects. If you don't include a column in ``dtypes``, the categories are inferred. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nrdtypesrcs0t|s i}||_||_t|||_dSr)rrrrr rr)rrrrrrrrs  zCategorizer.__init__r rr!csfddjDjjO_sSdtttfdtjfdd}jjfddt t d d |gd d dd ddS)Ncsg|] }|jvr|qSr)rr=rxr@rrr?sz$Categorizer._fit..unique_indicesr!cSst|Sr)rPCategoricalDtypekeys)rrrrcallbacksz"Categorizer._fit..callbackcst|dSr)r$r%)columns_to_getr rrr's z"Categorizer._fit..Tdrop_na_values)Zbase_fn callbackscSr(r)rr+rrrr'r-cSs|Srrr+rrrr'sr0) rr:rrrOrPrr2r3rr4)rr rr)rr rrr6s$  zCategorizer._fitr7cCs||j|j||j<|Sr)rZastyper:r)rr7rrrrRszCategorizer._transform_pandascCrS)NrTz , dtypes=rUr*)rrVrrrr@rrrrWs zCategorizer.__repr__)NN)rVrYrZr[r rOr rrPrrr r6r]rRrWr^rrrrr^s> rT)rrar rrr#rracs|duri}t}|D]}||vrtd|ddq dtjdtffdd dtjdttttfffd d }|j|d d }fd dD} |j ddD]3} | D],\} } | D]%} dd| D} | |vryt t |  || } | | | q_qYqS| S)NzYou set `max_categories` for z, which is not present in .r,r!csNt|rrtfdd}||S|dd}t|jddS)Ncs||Sr)update)r8rrrupdate_counters z\compute_unique_value_indices..get_pd_value_counts_per_column..update_countercSst|Sr)rD)r>rrrr'szVcompute_unique_value_indices..get_pd_value_counts_per_column..F)Zdropna)rGrrHZ value_countsto_dict)r,r)rrrget_pd_value_counts_per_columns  zDcompute_unique_value_indices..get_pd_value_counts_per_columnr7csJ|j}i}D]}||vr||g||<q td|d||S)NzColumn 'z2' does not exist in DataFrame, which has columns: )rrw ValueError)r7Z df_columnsresultr,)rrrrget_pd_value_countss z9compute_unique_value_indices..get_pd_value_countspandas)rcsi|]}|tqSr)set)r=r,r%rrrrz0compute_unique_value_indices..)Z batch_sizecSsi|] \}}|dur||qSrr)r=krkrrrrs)rrrPrQrr]rOr rZ iter_batchesrdictr most_commonrr)r rr#rraZ columns_setrxrZvalue_counts_dsZunique_values_by_colbatchr,Zcountersrr)rrrr#rr$s:&  r$Frr!cs"dtdtttfffdd }|S)a Returns a post-processing function that generates an encoding map by sorting the unique values produced during aggregation or stats computation. :param drop_na_values: If True, NA/null values will be silently dropped from the encoding map. If False, raises an error if any NA/null values are present. :return: A callable that takes a set of unique values and returns a dictionary mapping each value to a unique integer index. valuesr!csDr dd|D}n tdd|Drtdddtt|DS)NcSsh|] }t|s|qSrrPisnullr=rrrr sz:unique_post_fn..gen_value_index..css|]}t|VqdSrrrrrr "sz:unique_post_fn..gen_value_index..z]Unable to fit column because it contains null values. Consider imputing missing values first.cSrrr)r=jrrrrr'sz;unique_post_fn..gen_value_index..)anyr enumeratesorted)rrrrgen_value_indexsz'unique_post_fn..gen_value_index)r rrOr})rrrrrr4s  r4r7cs*fdd|D}|rtd|ddS)Ncs"g|] }|jr|qSr)rrrrr7rrr?-s"z _validate_df..zUnable to transform columns zJ because they contain null values. Consider imputing missing values first.)r)r7rZ null_columnsrrrrM,s  rMseriescCs4tdd|Dd}tjj|jot|ttj fS)Ncss|] }|dur|VqdSrr)r=r8rrrr8sz/_is_series_composed_of_lists..) nextrapitypesZis_object_dtyperrrnrNrorp)rZfirst_not_none_elementrrrrG5s  rG)F)+ collectionsr functoolsrtypingrrrrrr r r numpyrorrPZpandas.api.typesZ"ray.air.util.data_batch_conversionr Zray.data.preprocessorr rZray.data.preprocessors.utilsrZray.util.annotationsrZray.data.datasetrrr_r~rrrOr\r}r$r4r]rMrQrGrrrrsT (    !  w &B