B lnd&@sddlZddlmZddlmZmZmZmZddlmZ ddl m Z ddl m Z GdddZdd Zd d d hZdddZdddZdddZdS)N) _pandas_api)CodecTable concat_tablesschema)_feather) FeatherErrorc@s6eZdZdZd ddZd ddZdd Zdd d ZdS)FeatherDataseta Encapsulates details of reading a list of Feather files. Parameters ---------- path_or_paths : List[str] A list of file names validate_schema : bool, default True Check that individual file schemas are all the same / compatible TcCs||_||_dS)N)pathsvalidate_schema)selfZ path_or_pathsr r 3/tmp/pip-unpacked-wheel-i2z0tf32/pyarrow/feather.py__init__)szFeatherDataset.__init__NcCsnt|jd|d}|g|_|j|_x@|jddD].}t||d}|jrT||||j|q2Wt|jS)a, Read multiple feather files as a single pyarrow.Table Parameters ---------- columns : List[str] Names of columns to read from the file Returns ------- pyarrow.Table Content of the file as a table (of columns) r)columnsN) read_tabler Z_tablesrr validate_schemasappendr)r rZ_filpathtabler r rr-s  zFeatherDataset.read_tablecCs(|j|js$td||j|jdS)Nz-Schema in {!s} was different. {!s} vs {!s})requals ValueErrorformat)r Zpiecerr r rrFszFeatherDataset.validate_schemascCs|j|dj|dS)a Read multiple Parquet files as a single pandas DataFrame Parameters ---------- columns : List[str] Names of columns to read from the file use_threads : bool, default True Use multiple threads when converting to pandas Returns ------- pandas.DataFrame Content of the file as a pandas DataFrame (of columns) )r) use_threads)r to_pandas)r rrr r r read_pandasMs zFeatherDataset.read_pandas)T)N)NT)__name__ __module__ __qualname____doc__rrrrr r r rr s    r cCsN|jdkrdS|jttfkr4td|ntd|t|jdS)NrzqColumn '{}' exceeds 2GB maximum capacity of a Feather binary column. This restriction may be lifted in the futurezkColumn '{}' of type {} was chunked on conversion to Arrow and cannot be currently written to Feather format)Z num_chunkstypeextbinarystringrrstr)namecolr r rcheck_chunked_overflowas  r(lz4ZzstdZ uncompressedc Cstjr"tjr"t|tjjr"|}t|r|dkr:d}n|dkrHd}ntdt j ||d}|dkrx0t |j j D]\}} ||} t| | qtWn|}|dkrt|jtt|jkrtd|dk rtd|dk rtd n<|dkrtd rd }n$|dk r|tkrtd |tytj||||||d WnLtk rt|tr~yt|Wntjk r|YnXYnXdS)a Write a pandas.DataFrame to Feather format. Parameters ---------- df : pandas.DataFrame or pyarrow.Table Data to write out as Feather format. dest : str Local destination path. compression : string, default None Can be one of {"zstd", "lz4", "uncompressed"}. The default of None uses LZ4 for V2 files if it is available, otherwise uncompressed. compression_level : int, default None Use a compression level particular to the chosen compressor. If None use the default compression level chunksize : int, default None For V2 files, the internal maximum size of Arrow RecordBatch chunks when writing the Arrow IPC file format. None means use the default, which is currently 64K version : int, default 2 Feather file version. Version 2 is the current. Version 1 is the more limited legacy format rFr*Nz%Version value should either be 1 or 2)preserve_indexz'cannot serialize duplicate column namesz2Feather V1 files do not support compression optionz0Feather V1 files do not support chunksize optionZ lz4_framer)z1compression="{}" not supported, must be one of {}) compressioncompression_level chunksizeversion)rZ have_pandasZ has_sparse isinstancepdZSparseDataFrameZto_denseZ is_data_framerrZ from_pandas enumeraternamesr(lenZ column_namessetrZ is_available_FEATHER_SUPPORTED_CODECSrr write_feather Exceptionr%osremoveerror) Zdfdestr,r-r.r/r+rir&r'r r rr7ssR      r7TFcKs t||||djfd|i|S)a Read a pandas.DataFrame from Feather format. To read as pyarrow.Table use feather.read_table. Parameters ---------- source : str file path, or file-like object You can use MemoryMappedFile as source, for explicitly use memory map. columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. use_threads : bool, default True Whether to parallelize reading using multiple threads. If false the restriction is used in the conversion to Pandas as well as in the reading from Feather format. memory_map : boolean, default False Use memory mapping when opening file on disk, when source is a str. **kwargs Additional keyword arguments passed on to `pyarrow.Table.to_pandas`. Returns ------- df : pandas.DataFrame The contents of the Feather file as a pandas.DataFrame )r memory_maprr)rr)sourcerrr>kwargsr r r read_feathersrAcCstj|||d}|dkr |Sdd|D}ttdd|rL||}nszread_table..cSs|tkS)N)int)tr r rzread_table..cSs|tkS)N)r%)rFr r rrGrHcSsg|] }|jqSr )r)rBrFr r rrDszrreaderZ column_typesrZcolumn_type_namesr r rrs"     r)NNNr*)NTF)NFT)r9Zpyarrow.pandas_compatrZ pyarrow.librrrrlibr"ZpyarrowrZpyarrow._featherrr r(r6r7rArr r r rs    D  S