o 1 i @sddlZddlmZddlmZmZmZddlZddl m Z m Z m Z m Z mZmZmZmZmZer6ddlmZeeZdedee fdd Zdedee fd d Zdedee fd d ZeGdddZ ddddeeedefddZdS)N) dataclass) TYPE_CHECKINGListOptional) AggregateFnV2ApproximateQuantileCountMaxMeanMinMissingValuePercentageStdZeroPercentage)Datasetcolumnreturnc CsVt|ddt|ddt|ddt|ddt|dddt|dgdt|dt|ddgS) aGenerate default metrics for numerical columns. This function returns a list of aggregators that compute the following metrics: - count - mean - min - max - std - approximate_quantile - missing_value_percentage - zero_percentage Args: column: The name of the numerical column to compute metrics for. Returns: A list of AggregateFnV2 instances that can be used with Dataset.aggregate() Fon ignore_nullsTr)rrZddofg?)rZ quantilesr)rr r r r rr rrrZ/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/data/stats.pynumerical_aggregatorss       rcCt|ddt|dgS)agGenerate default metrics for string columns. This function returns a list of aggregators that compute the following metrics: - count - MissingValuePercentage Args: column: The name of the categorical column to compute metrics for. Returns: A list of AggregateFnV2 instances that can be used with Dataset.aggregate() Frrrr rrrrcategorical_aggregators9s rcCr)zGenerate default metrics for vector columns. This function returns a list of aggregators that compute the following metrics: - count - MissingValuePercentage Frrrrrrrvector_aggregatorsLs rc@sBeZdZUdZeeed<eeed<eeed<eeed<dS)FeatureAggregatorsz8Container for categorized columns and their aggregators.numerical_columns str_columnsvector_columns aggregatorsN)__name__ __module__ __qualname____doc__rstr__annotations__rrrrrrYs    rdatasetrcolumnsc Csr|}|s td|dur|j}t|t|j}|r$td|dg}g}g}g}|j}|j} tt|| } |D]u} | | vrBq;| | } t| tj sYt d| d| dq;tj | sqtj | sqtj| sqtj| r~|| |t| q;tj| r|| |t| q;tj| r|| |t| q;t d| d| dq;t||||d S) a1Generate aggregators for all columns in a dataset. Args: dataset: A Ray Dataset instance columns: A list of columns to include in the summary. If None, all columns will be included. Returns: FeatureAggregators containing categorized column names and their aggregators z9Dataset must have a schema to determine numerical columnsNzColumns z not found in dataset schemazSkipping field z: type z is not a PyArrow DataTypez not supported)rr r!r")schema ValueErrornamessettypesdictzip isinstancepaZDataTypeloggerwarning is_integerZ is_floatingZ is_decimalZ is_booleanappendextendrZ is_stringrZis_listrr) r)r*r+Z missing_colsrr r!Zall_aggsZ column_namesZ column_typesZ name_to_typenameZftyperrrfeature_aggregators_for_datasetcs^           r:)N)logging dataclassesrtypingrrrZpyarrowr3Zray.data.aggregaterrrr r r r r rZray.datar getLoggerr#r4r'rrrrr:rrrrs* ,