o 1 iKl @sddlZddlZddlZddlmZddlmZmZmZm Z ddl m Z er*ddl Z dZ dZeeZeaddZd d Zd d Zd dZdfddZdee eddfddddfddZddde eddffddZd edd!dddfd"d#ZeGd$ddZdgd&d'Zdhd)d*Zd!ddefd+d,Z d!ddefd-d.Z!did0d1Z"dhd2d3Z#dhd4d5Z$dhd6d7Z% ( 8  djd9d:Z&dkdzddl}Wn tyYdSwt|t|t|dS)Nr)pyarrowModuleNotFoundError_register_arrow_data_serializer+_register_arrow_json_readoptions_serializer,_register_arrow_json_parseoptions_serializerserialization_contextparl/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/_private/arrow_serialization.py%_register_custom_datasets_serializerss   rcDtjtddkr dSddlm|jjddfddddS)N01rcSs |j|jfSN)Z use_threads block_sizeoptsrrr:s z=_register_arrow_json_readoptions_serializer..c j|Sr) ReadOptionsargsZpajsonrrr; Zcustom_serializerZcustom_deserializer)osenvirongetr pyarrow.jsonjson _register_cloudpickle_serializerrrrr!rr ,s   r cr)NrrrcSs|j|j|jfSr)Zexplicit_schemaZnewlines_in_valuesZunexpected_field_behaviorrrrrrMsz>_register_arrow_json_parseoptions_serializer..crr) ParseOptionsrr!rrrRr"r#)r$r%r&rr'r(r)r+r*rr!rr?s   rcCs0tjtddkr dSddl}||jtdS)ayCustom reducer for Arrow data that works around a zero-copy slicing pickling bug by using the Arrow IPC format for the underlying serialization. Background: Arrow has both array-level slicing and buffer-level slicing; both are zero-copy, but the former has a serialization bug where the entire buffer is serialized instead of just the slice, while the latter's serialization works as expected and only serializes the slice of the buffer. I.e., array-level slicing doesn't propagate the slice down to the buffer when serializing the array. We work around this by registering a custom cloudpickle reducers for Arrow Tables that delegates serialization to the Arrow IPC format; thankfully, Arrow's IPC serialization has fixed this buffer truncation bug. See https://issues.apache.org/jira/browse/ARROW-10739. rrNr)r$r%r&r r Z_register_cloudpickle_reducerTable_arrow_table_reducerrrrr Xsr t pyarrow.Tablec Csg}|jD]S}||}zt|}WnAtyR}z5t|js$tr$|dt|jtvrBtjd|d|jdddt t|jt |WYd}~Sd}~ww| |qt ||j ffS)a/Custom reducer for Arrow Tables that works around a zero-copy slice pickling bug. Background: Arrow has both array-level slicing and buffer-level slicing; both are zero-copy, but the former has a serialization bug where the entire buffer is serialized instead of just the slice, while the latter's serialization works as expected and only serializes the slice of the buffer. I.e., array-level slicing doesn't propagate the slice down to the buffer when serializing the array. All that these copy methods do is, at serialization time, take the array-level slicing and translate them to buffer-level slicing, so only the buffer slice is sent over the wire instead of the entire buffer. See https://issues.apache.org/jira/browse/ARROW-10739. NzTFailed to complete optimized serialization of Arrow Table, serialization of column 'z ' of type z failed, so we're falling back to Arrow IPC serialization for the table. Note that this may result in slower serialization and more worker memory utilization. Serialization error:T)exc_info)Z column_names_arrow_chunked_array_reduce Exception_is_dense_uniontyper_serialization_fallback_setloggerwarningadd_arrow_table_ipc_reduceappend_reconstruct_tableschema)r.reduced_columnsZ column_namecolumnZreduced_columnerrrr-qs.   r-r= pyarrow.Arraypyarrow.DataTyper<zpyarrow.SchemareturncCs:ddl}g}|D] \}}|t||q|jj||dS)zERestore a serialized Arrow Table, reconstructing each reduced column.rNr<)r r:_reconstruct_chunked_arrayr, from_arrays)r=r<rcolumnsZchunks_payloadtype_rrrr;s  r;cazpyarrow.ChunkedArrayPicklableArrayPayloadcCs.g}|jD] }t|}||q||jfS)zCustom reducer for Arrow ChunkedArrays that works around a zero-copy slice pickling bug. This reducer does not return a reconstruction function, since it's expected to be reconstructed by the Arrow Table reconstructor. )chunksrI from_arrayr:r4)rHZchunk_payloadschunkZ chunk_payloadrrrr1s     r1rJrGcCs"ddl}dd|D}|||S)z=Restore a serialized Arrow ChunkedArray from chunks and type.rNcSg|]}|qSrto_array).0rLrrr z._reconstruct_chunked_array..)r Z chunked_array)rJrGrrrrrDs rDc@sbeZdZUdZded<eed<eded<eed<eed<eded <edd dZdddZ dS)rIzPicklable array payload, holding data buffers and array metadata. This is a helper container for pickling and reconstructing nested Arrow Arrays while ensuring that the buffers that underly zero-copy slice views are properly truncated. rAr4lengthpyarrow.Bufferbuffers null_countoffsetchildrenar@rBcCst|S)a Create a picklable array payload from an Arrow Array. This will recursively accumulate data buffer and metadata payloads that are ready for pickling; namely, the data buffers underlying zero-copy slice views will be properly truncated. )_array_to_array_payload)selfrYrrrrKsz PicklableArrayPayload.from_arraycCst|S)z7Reconstruct an Arrow Array from this picklable payload.)_array_payload_to_array)r[rrrrOszPicklableArrayPayload.to_arrayNrYr@rBrI)rBr@) __name__ __module__ __qualname____doc____annotations__intr classmethodrKrOrrrrrIs     payloadc Csddl}dd|jD}|j|jr*t|dksJt||\}}|j||S|j|jrPt|dkrPt|dksCJt||\}}}|j |||St |j|j rmt|dkscJt||d}|j |S|j j|j|j|j|j|j|dS) zHReconstruct an Arrow Array from a possibly nested PicklableArrayPayload.rNcSrMrrN)rPZ child_payloadrrrrQrRz+_array_payload_to_array..r4rSrUrVrWrX)r rXtypes is_dictionaryr4lenZDictionaryArrayrEis_mapZMapArray isinstanceBaseExtensionTypeZ wrap_arrayArray from_buffersrSrUrVrW) rerrXindices dictionaryoffsetskeysitemsstoragerrrr\s,  r\rYcCsddl}t|jr td|j|jrt|St|jr!t|St |jr*t |S|j |js8|j |jr.ri) rUrlrVrrWranger4 num_fieldsrI)rYrUrrXrrrrs rpyarrow.UnionArraycsddl}tjr J}t|dksJt||d}|dus'J||d}t||jt}fddtjj D}t jt||gj d|dS)z0Serialize union arrays to PicklableArrayPayload.rNrgcrrrrrrrrQrz1_union_array_to_array_payload..ri) r r3r4rUrlrZint8rWrrrIrV)rYrrUrZ type_code_bufrXrrrrs"rpyarrow.DictionaryArraycCs4t|j}t|j}t|jt|g|jd||gdS)z5Serialize dictionary arrays to PicklableArrayPayload.rri)rZrrrsrIr4rlrV)rYZindices_payloadZdictionary_payloadrrrrs  rpyarrow.MapArrayc Cs4ddl}|}t|dksJt||jdkr%t|d|jt|}nd}|g}|d}t||j|jt|\}}}t||j j rS| |t |j ||g}n9|}t|dkscJt||j|t|d||g} |j||} |j||} t | t | t | g}t|jt|||jd|dS)z.Serialize map arrays to PicklableArrayPayload.rNrgrfri)r rUrlrVrrWrr4rnlibZ ListArrayr:rZrrrprqint32rurvrI) rYrrUrZ new_buffersrrrrXrtrurvrrrr/sB   rpyarrow.ExtensionArraycCs(t|j}t|jt|g|jd|gdS)Nrri)rZrwrIr4rlrV)rYZstorage_payloadrrrrgs rbufrTrWrScCsTddl}|dur|j|rt|||}|S|dur|jdnd}t||||}|S)Copy buffer, if needed.rNrg)r rjrrZ bit_width_copy_normal_buffer_if_needed)rrGrWrSrZtype_bytewidthrrrrus r byte_widthcCs2||}||}|dks||jkr|||}|S)rr)sizer)rrrWrS byte_offset byte_lengthrrrrs  rcCsV|d}|d}t||d}|dks||jkr)|||}|dkr)t|||}|S)z)Copy bit-packed binary buffer, if needed.rr)_bytes_for_bitsrr_align_bit_offset)rrWrS bit_offsetrrrrrrs  rarr_typec Csddl}ddlm}|j|s"|j|s"|j|s"|j|r'|}n| }t ||||d}|j ||dd|g}|d }|d |} |||}|j|ra|j|dd}|d}||| fS)zvCopy the provided offsets buffer, returning the copied buffer and the offset + length of the underlying data. rNrgF)safe)r Zpyarrow.computeZcomputerjr~rrZis_large_unicodeZint64rrrprqZas_pysubtractZis_int32castrU) rrrWrSrZpacZ offset_typertrrrrrrs*           rncCs |dd@S)zpRound up n to the nearest multiple of 8. This is used to get the byte-padded number of bits for n bits. ir)rrrrrs rrrcCs>ddl}|}t|tj}||L}||tj}||S)z}Align the bit offset into the buffer with the front of the buffer by shifting the buffer and eliminating the offset. rN)r Z to_pybytesrc from_bytessys byteorderto_bytesZ py_buffer)rrrrbytes_Z bytes_as_intrrrrs  rtablecCsdddlm}ddlm}|}|||jd }||Wdn1s&wYt|ffS)a4Custom reducer for Arrow Table that works around a zero-copy slicing pickling bug by using the Arrow IPC format for the underlying serialization. This is currently used as a fallback for unsupported types (or unknown bugs) for the manual buffer truncation workaround, e.g. for dense unions. r)RecordBatchStreamWriter)BufferOutputStreamrCN) pyarrow.ipcrZ pyarrow.librr<Z write_table_restore_table_from_ipcgetvalue)rrrZ output_streamwrrrrr9s   r9cCs@ddlm}|| }|WdS1swYdS)z6Restore an Arrow Table serialized to Arrow IPC format.r)RecordBatchStreamReaderN)rrZread_all)rrreaderrrrrs  $rcCsddl}|j|o|jdkS)z1Whether the provided Arrow type is a dense union.rNZdense)r rjrmoderrrrr3sr3)r.r/)rerIrBr@r])rYrrBrI)rYrrBrI)rYrrBrI)rYrrBrI)rYrrBrI)rYrrBrI)rYrrBrI)rr/)7loggingr$r dataclassesrtypingrrrrZray._private.utilsrr rr getLoggerr^r6setr5rr rr r-r;r1rDrIr\rZboolrzr|ryr{r}rrrrrrrrcrrrrrrr9bytesrr3rrrrs    .     # ")  "  ! 8      %