o i@sddlmZddlmZddlmZmZmZmZddl m Z ddl m Zer4ddlmZddlmZmZddd d&ddZe d'dddd(ddZe d'dd d)d#dZ d*dddd+d%dZdS),) annotations)partial) TYPE_CHECKINGAnyLiteraloverloadNpyarrow)Iterator) DataFrame LazyFrameT)allow_pyarrow_filter batch_sizedspa.dataset.Datasetr boolr int | Nonereturnr cCs:|rdnd}tt|fd|i||i}tjj|j||dS)a' Pickle the partially applied function `_scan_pyarrow_dataset_impl`. The bytes are then sent to the polars logical plan. It can be deserialized once executed and ran. Parameters ---------- ds pyarrow dataset allow_pyarrow_filter Allow predicates to be pushed down to pyarrow. This can lead to different results if comparisons are done with null values as pyarrow handles this different than polars does. batch_size The maximum row count for scanned pyarrow record batches. ruser_batch_sizer r)r_scan_pyarrow_dataset_implplr Z_scan_python_functionZschema)rr rZbatch_size_keyfuncr_/home/app/Keep/.python/lib/python3.10/site-packages/polars/io/pyarrow_dataset/anonymous_scan.py_scan_pyarrow_datasets r.)r r with_columnslist[str] | None predicatestr | bytes | Nonen_rows Literal[True]rr cCdSNrrrrrrr rrrrr4 r)rLiteral[False] tuple[Iterator[DataFrame], bool]cCr!r"rr#rrrrAr$,DataFrame | tuple[Iterator[DataFrame], bool]c sd}d|r7|dur7ddlm}m} m} m} ddlm} m} m}t |t | | ||| | | d}dur5|}n|||d|durB|n|}|durL|d<d fd d }|r^|\}|S|d fS)a Take the projected columns and materialize an arrow table. Parameters ---------- ds pyarrow dataset. with_columns Columns that are projected. predicate pyarrow expression string (when `allow_pyarrow_filter=True`) or serialized Polars predicate bytes (when `allow_pyarrow_filter=False`). n_rows: Materialize only `n` rows from the arrow dataset. batch_size The maximum row count for scanned pyarrow record batches. allow_pyarrow_filter If True, evaluate predicate and return DataFrame directly. If False, return `(generator, False)` tuple for IOPlugin path. user_batch_size User-specified `batch_size` (takes precedence over Rust-provided `batch_size`). Warnings -------- Don't use this if you accept untrusted user inputs. Predicates will be evaluated with python 'eval'. There is sanitation in place, but it is a possible attack vector. Returns ------- DataFrame or tuple[Iterator[DataFrame], bool] Nr) to_py_dateto_py_datetime to_py_timeto_py_timedelta)DateDatetimeDuration)par,r-r.r(r)r*r+)columnsfilterrrIterator[DataFrame]c3sXtdur durjfinjfinjdiVdS)Nr)rr headr1Zto_tablerZ common_paramsrZfilter_post_slice_rrrframess z*_scan_pyarrow_dataset_impl..framesF)rr2) Zpolars._utils.convertr(r)r*r+Zpolars.datatypesr,r-r.evalr/)rrrrrr rfilter_r(r)r*r+r,r-r.vr5xrr4rrNs:*    )rrr rrrrr ).)rrrrrrrrrrr r rrrr )rrrrrrrrrrr r%rrrr&r")rrrrrrrrrrr rrrrr') __future__r functoolsrtypingrrrrZpolars._reexportZ _reexportrZpolars._dependenciesr r/collections.abcr Zpolarsr r rrrrrrs6    %