o 1 i*@sddlmZmZmZmZmZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZGdddZeGd d d eZeGd d d ZGd ddeZeGdddegee fZeGdddeZdS))CallableDictIterableListOptionalN)_check_pyarrow_version)Block BlockMetadataSchema)_iter_sliced_blocks) Deprecated DeveloperAPI PublicAPIc@s^eZdZdZdefddZdeeefddZ deeedee eefdd fd d Z d S) "_DatasourceProjectionPushdownMixinz:Mixin for reading operators supporting projection pushdownreturncCdS)zvReturns ``True`` in case ``Datasource`` supports projection operation being pushed down into the reading layerFselfrrj/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/data/datasource/datasource.pysupports_projection_pushdownz?_DatasourceProjectionPushdownMixin.supports_projection_pushdowncCr)zRetrurns current projectionNrrrrrget_current_projectionsz9_DatasourceProjectionPushdownMixin.get_current_projectioncolumnscolumn_rename_map DatasourcecCs|SNr)rrrrrrapply_projectionz3_DatasourceProjectionPushdownMixin.apply_projectionN) __name__ __module__ __qualname____doc__boolrrrstrrrrrrrrr s rc@seZdZdZedddZedededfdd Zde fd d Z de efd d Z ddede ededfddZ edefddZedefddZdS)rzInterface for defining a custom :class:`~ray.data.Dataset` datasource. To read a datasource into a dataset, use :meth:`~ray.data.read_datasource`. rReadercKst|fi|Sz Deprecated: Implement :meth:`~ray.data.Datasource.get_read_tasks` and :meth:`~ray.data.Datasource.estimate_inmemory_data_size` instead. )_LegacyDatasourceReader)r read_argsrrr create_reader&szDatasource.create_reader parallelismReadTaskcKtr&NotImplementedError)rr*r(rrr prepare_read.szDatasource.prepare_readcCs.t|j}d}||r|dt| }|S)ztReturn a human-readable name for this datasource. This will be used as the names of the read tasks. rN)typerendswithlen)rnameZdatasource_suffixrrrget_name6s  zDatasource.get_namecCr,zReturn an estimate of the in-memory data size, or None if unknown. Note that the in-memory data size may be larger than the on-disk data size. r-rrrrestimate_inmemory_data_size@rz&Datasource.estimate_inmemory_data_sizeNper_task_row_limitcCr,)aExecute the read and return read tasks. Args: parallelism: The requested read parallelism. The number of read tasks should equal to this value if possible. per_task_row_limit: The per-task row limit for the read tasks. Returns: A list of read tasks that can be executed to read blocks from the datasource in parallel. r-rr*r7rrrget_read_tasksGs zDatasource.get_read_taskscCs,t|jtju}t|jtju}| p| Sr)r0r9rr6)rZhas_implemented_get_read_tasksZ+has_implemented_estimate_inmemory_data_sizerrrshould_create_readerVszDatasource.should_create_readercCr)z:If ``False``, only launch read tasks on the driver's node.Trrrrrsupports_distributed_readsdrz%Datasource.supports_distributed_reads)rr%r)rr r!r"r r)intrr/r$r4rr6r9propertyr#r:r;rrrrrs(    rc@s8eZdZdZdeefddZdededfddZd S) r%aA bound read operation for a :class:`~ray.data.Datasource`. This is a stateful class so that reads can be prepared in multiple stages. For example, it is useful for :class:`Datasets ` to know the in-memory size of the read prior to executing it. rcCr,r5r-rrrrr6srz"Reader.estimate_inmemory_data_sizer*r+cCr,)aMExecute the read and return read tasks. Args: parallelism: The requested read parallelism. The number of read tasks should equal to this value if possible. Returns: A list of read tasks that can be executed to read blocks from the datasource in parallel. r-)rr*rrrr9zs zReader.get_read_tasksN) rr r!r"rr<r6rr9rrrrr%jsr%c@sNeZdZdefddZdeefddZ d ded eeded fd d Z dS)r' datasourcecKs||_||_dSr) _datasource _read_args)rr>r(rrr__init__s z _LegacyDatasourceReader.__init__rcCsdSrrrrrrr6sz3_LegacyDatasourceReader.estimate_inmemory_data_sizeNr*r7r+cCs|jj|fi|jS)aExecute the read and return read tasks. Args: parallelism: The requested read parallelism. The number of read tasks should equal to this value if possible. per_task_row_limit: The per-task row limit for the read tasks. Returns: A list of read tasks that can be executed to read blocks from the datasource in parallel. )r?r/r@r8rrrr9sz&_LegacyDatasourceReader.get_read_tasksr) rr r!rrArr<r6rr9rrrrr'sr'c @seZdZdZ  ddegeefdededdee fdd Z e d efd d Z e d edfd dZ e d egeeffddZe d ee fddZd eefddZdS)r+aA function used to read blocks from the :class:`~ray.data.Dataset`. Read tasks are generated by :meth:`~ray.data.Datasource.get_read_tasks`, and return a list of ``ray.data.Block`` when called. Initial metadata about the read operation can be retrieved via the ``metadata`` attribute prior to executing the read. Final metadata is returned after the read along with the blocks. Ray will execute read tasks in remote functions to parallelize execution. Note that the number of blocks returned can vary at runtime. For example, if a task is reading a single large file it can return multiple blocks to avoid running out of memory during the read. The initial metadata should reflect all the blocks returned by the read, e.g., if the metadata says ``num_rows=1000``, the read can return a single block of 1000 rows, or multiple blocks with 1000 rows altogether. The final metadata (returned with the actual block) reflects the exact contents of the block itself. Nread_fnmetadataschemar r7cCs||_||_||_||_dSr) _metadata_read_fn_schema_per_task_row_limit)rrBrCrDr7rrrrAs zReadTask.__init__rcC|jSr)rErrrrrCzReadTask.metadatacCrIr)rGrrrrrDrJzReadTask.schemacCrIr)rFrrrrrBrJzReadTask.read_fncCrI)z.Get the per-task row limit for this read task.)rHrrrrr7szReadTask.per_task_row_limitccsP|}t|dstd||jdur|EdHdSt||jEdHdS)N__iter__zlRead function must return Iterable[Block], got {}. Probably you need to return `[block]` instead of `block`.)rFhasattrDeprecationWarningformatrHr )rresultrrr__call__s   zReadTask.__call__)NN)rr r!r"rrrr rr<rAr=rCrDrBr7rPrrrrr+s,  r+c@sdeZdZdZdedefddZdeefddZ dd ed eedee fd d Z de fddZ d S)RandomIntRowDatasourceaAn example datasource that generates rows with random int64 columns. Examples: >>> import ray >>> from ray.data.datasource import RandomIntRowDatasource >>> source = RandomIntRowDatasource() # doctest: +SKIP >>> ray.data.read_datasource( # doctest: +SKIP ... source, n=10, num_columns=2).take() {'c_0': 1717767200176864416, 'c_1': 999657309586757214} {'c_0': 4983608804013926748, 'c_1': 1160140066899844087} n num_columnscCs||_||_dS)zInitialize the datasource that generates random-integer rows. Args: n: The number of rows to generate. num_columns: The number of columns to generate. N_n _num_columns)rrRrSrrrrAs zRandomIntRowDatasource.__init__rcCs|j|jdS)NrTrrrrr6sz2RandomIntRowDatasource.estimate_inmemory_data_sizeNr*r7c stddlg}|j}|j}td||}dtdtdtffdd jdd t |Dj }d}||krdt |||} t | d | |ddd } | t| |ffd d | ||d||7}||ks7|S)NrcountrSrcs<jjtjjttjj||ftjdddt|DdS)N)sizeZdtypecSsg|]}d|qS)c_r.0irrr szMRandomIntRowDatasource.get_read_tasks..make_block..)names) TableZ from_arraysnprandomrandintZiinfoZint64maxrangerYrS)pyarrowrr make_block s z9RandomIntRowDatasource.get_read_tasks..make_blockcSsi|] }d|dgqS)r[rrr\rrr sz9RandomIntRowDatasource.get_read_tasks..rW)Znum_rowsZ size_bytesZ input_filesZ exec_statscs ||gSrrrg)rirr#sz7RandomIntRowDatasource.get_read_tasks..)rDr7)rrhrUrVrer<rraZ from_pydictrfrDminr appendr+) rr*r7Z read_tasksrRrS block_sizerDr^rYmetar)rirhrr9s@  z%RandomIntRowDatasource.get_read_taskscCr)zReturn a human-readable name for this datasource. This will be used as the names of the read tasks. Note: overrides the base `Datasource` method. Z RandomIntrrrrrr4/rzRandomIntRowDatasource.get_namer) rr r!r"r<rArr6rr+r9r$r4rrrrrQs   0rQ)typingrrrrrnumpyrbZray.data._internal.utilrZray.data.blockrr r Zray.data.datasource.utilr Zray.util.annotationsr r rrrr%r'r+rQrrrrs   JB