o * i[O @sUddlmZddlZddlmZmZmZddlZddl Z ddl m Z ddl m Z ddl m Z mZdd lmZdd lmZmZmZmZmZmZmZermdd lmZdd lmZmZdd lmZee ee!dfZ"de#d<gZ$iej%j&j'dej%j&j(dej%j&j)dej%j&j*dej%j&j+dej%j&j,dej%j&j-dej%j&j.dej%j&j/dej%j&j0dej%j&j1dej%j&j2dej%j&j3dej%j&j4dej%j&j5dej%j&j6d Z7ej%j&j'ej%j&j*ej%j&j+ej%j&j,ej%j&j-ej%j&j.ej%j&j/ej%j&j0ej%j&j1ej%j&j2ej%j&j3ej%j&j4d! Z8dFd#d$Z9d%d&Z:dGd+d,Z; -dHd.d/Ze?e!ee fe@ee fd4fdId>d?ZAGd@dAdAZBGdBdCdCZCGdDdEdEZDdS)J) annotationsN) TYPE_CHECKINGAnyUnion)pir)Value)_PADDLE_PIR_DTYPE_2_NUMPY_DTYPE ParameterMeta)core)EagerParamBaseVariable_cpu_num _cuda_idsdefault_main_programin_dygraph_mode in_pir_mode) TypeAlias) DTypeLike ShapeLike) _DTypeLiteral) _ClassInfo.rrbool float8_e4m3fn float8_e5m2float16uint16float32float64int8int16int32int64uint8 complex64 complex128Zpstringraw) rrrrrr r!r"r#r$r%r&NCHWcCst|jdkr |tjS|dkrt|gd}tjddtjgd|j}t||j}|dkr8t|gd}|S)NrZNHWC)rr rcSstdtd|dd?S)N[z)convert_float_to_uint16..Zotypes)rrr)r ) sizeviewnprZ transpose vectorizeflatreshapeshape)dataZ data_formatnew_datar3r3r4convert_float_to_uint16Ss  rAcCs*tjddtjgd|j}t||jS)NcSstdtd|d>dS)Nr+r*r,rr-r1r3r3r4r5gr6z)convert_uint16_to_float..r7)r:r;rr<r=r>)r?r@r3r3r4convert_uint16_to_floatesrBdtyperreturnrc Cst|tjjr|tvrt|St|tjr|tvrt|Sn2t|trB|tt j t j t j t j t jt jt jt jt jt jt jf vrA|jSn|dvrJt|S|dvrPdStd|)N)rrrrrZint4r r!r"r#r$r%r&rr)Zbfloat16rzdtype must be any of [bool, float16, uint16, float32, float64, int8, int16, int32, int64, uint8, complex64, complex128, bfloat16], but received ) isinstancer VarDescVarType_PADDLE_DTYPE_2_NUMPY_DTYPEZDataTyper typerr:rrrrr r!r"r#r$r%r&__name__str TypeErrorrCr3r3r4 convert_dtypemsB  rNcCsjtr ddlm}|jrtttf}nttf}t|||||n t||ttf||t |j ||||dS)Nrlazy_init_helper) rnn.initializer.lazy_initrQstaterr r check_typer check_dtyperC)input input_nameexpected_dtypeop_name extra_messagerQexpectedr3r3r4check_variable_and_dtypes  r\c CstrdSddlm}ddlm}|r$t|ts|f}|tjj f7}nt|tjj r:|j s:t d|d|dt||sTt d|d|d |d t |d | dS) NrrPr )in_to_static_modezPlease use `with base.dygraph.guard()` as context or `paddle.disable_static()` to switch to dygraph mode firstly. Because received '' in z is an Eager Tensor.z The type of ' must be , but received . ) rrRrQZ dygraph.baser]rEtupler eagerZTensorrSrLrI)rVrW expected_typerYrZrQr]r3r3r4rTs*    $rTc CsDtrdSt||vr td|d|d|dt|d| dS)NzThe data type of 'r^r_r`ra)rrNrL)Z input_dtyperWrXrYrZr3r3r4rUs $rU)r"r#r>rrYrKexpected_shape_typeexpected_element_typeexpected_tensor_dtypetuple[_DTypeLiteral, ...]Nonec CstrdSt|d|||dur=t|ttfs=|D]#}t|d|||durzelement of shapez9If element of shape is Tensor, its data type should be {}z, ) rrTrErrrUrCformatjoin)r>rYrerfrgitemr3r3r4 check_shapes4   rmc@s<eZdZddZddZddZddZd d Zd d Zd S)DataToDenseTensorConvertercCsZ||_||_||_d}|jD]}|dkr|d7}|dkr!d|_nqt||_|dSNrr )place lod_levelr>rNrC_reset)selfrprqr>rCZnegative_countsr3r3r4__init__s   z#DataToDenseTensorConverter.__init__cCs g|_ddt|jD|_dS)NcSsg|]}gqSr3r3).0_r3r3r4 "sz5DataToDenseTensorConverter._reset..)r?rangerqlodrsr3r3r4rr sz!DataToDenseTensorConverter._resetcCs|||j|jdSN) _feed_impl_rzrq)rsr?r3r3r4feed$szDataToDenseTensorConverter.feedcCsR|dkr |j|dS|dt||D]}|||dd|dqdSro)r?appendlenr})rsr?rzrqZ each_datar3r3r4r}'s z&DataToDenseTensorConverter._feed_impl_cCsHt|j|D]\}}||kr!|dkr!|dkr!td|jd|qdS)Nrz2Shape not match. What is defined in data layer is , but receive )zipr> ValueError)rsr>s1s2r3r3r4 _check_shape/sz'DataToDenseTensorConverter._check_shapecCstj|j|jd}|jr2t|jt|jkr2z||j}Wnty1td|jd|jwt }| ||j |j dkrH| |j||S)NrMz0Reshape error. What is defined in data layer is rr)r:arrayr?rCr>rr=rr Z DenseTensorsetrprqZset_recursive_sequence_lengthsrzrr)rsZarrtr3r3r4done6s    zDataToDenseTensorConverter.doneN) rJ __module__ __qualname__rurrr~r}rrr3r3r3r4rns rnc@s$eZdZddZddZddZdS)BatchedTensorProviderc Csb||_||_||_g|_||_|D]}ts|jdksJd|jt|jd|j |j dqdS)Nrzlod_level must be 0rprqr>rC) rp batch_size generator converters drop_lastrrqrrnr>rC)rs feed_listrprrrvarr3r3r4ruIs"zBatchedTensorProvider.__init__cCsdd|jDS)NcSg|]}|qSr3)rrvcr3r3r4rx]z/BatchedTensorProvider._done..)rr{r3r3r4_done\szBatchedTensorProvider._doneccsd}|D]#}t||jD] \}}|j|q|d7}||jkr*d}|Vq|js9|dkr9|VdSdd|jDdS)Nrr cSrr3)rrrr3r3r4rxmrz2BatchedTensorProvider.__call__..)rrrr?rrrr)rsidx each_sample each_sloteach_converterr3r3r4__call___s   zBatchedTensorProvider.__call__N)rJrrrurrr3r3r3r4rHs rc@s*eZdZdZd ddZddZddZdS) DataFeedera :api_attr: Static Graph DataFeeder converts the data that returned by a reader into a data structure that can feed into Executor. The reader is usually a python generator that returns a list of mini-batch data entries. Parameters: feed_list (list): Variables or names of Variables that need to feed. place (:ref:`api_paddle_CPUPlace` | :ref:`api_paddle_CUDAPlace` ): place indicates the device (CPU | GPU) the data will be fed into, if you want to feed data into GPU, please using :code:`base.CUDAPlace(i)` (:code:`i` represents the GPU id), or if you want to feed data into CPU, please using :code:`base.CPUPlace()`. program (:ref:`api_paddle_static_Program` , optional): The Program that will feed data into, if program is None, it will use default_main_program(). Default None. Raises: :code:`ValueError` - If some Variables are not in this Program. Example: .. code-block:: python >>> import numpy as np >>> import paddle >>> from paddle import base >>> paddle.enable_static() >>> place = paddle.CPUPlace() >>> def reader(): ... for _ in range(4): ... yield np.random.random([4]).astype('float32'), np.random.random([3]).astype('float32'), ... >>> main_program = paddle.static.Program() >>> startup_program = paddle.static.Program() >>> with paddle.static.program_guard(main_program, startup_program): ... data_1 = paddle.static.data(name='data_1', shape=[None, 2, 2], dtype='float32') ... data_2 = paddle.static.data(name='data_2', shape=[None, 1, 3], dtype='float32') ... out = paddle.static.nn.fc(x=[data_1, data_2], size=2) ... # ... >>> feeder = base.DataFeeder([data_1, data_2], place) >>> exe = paddle.static.Executor(place) >>> exe.run(startup_program) >>> feed_data = feeder.feed(reader()) >>> # print feed_data to view feed results >>> # print(feed_data['data_1']) >>> # print(feed_data['data_2']) >>> outs = exe.run( ... program=main_program, ... feed=feed_data, ... fetch_list=[out] ... ) >>> print(outs) NcCs"g|_g|_g|_g|_||_trO|durtj}|D]/}t |t r(t dt |t s1t d|j|j|j|j|jd|j|jqdS|durVt}|D]6}t |t rg|d|}t |tt fsrt d|j|j|j|j|j|j|j|jqXdS)Nz+In PIR Mode, Not supported string input yetz(Feed list should contain a list of Valuerz+Feed list should contain a list of variable) feed_dtypes feed_names feed_shapesfeed_lod_levelrprrr rrErKrrrLrrCnamer>blockrrrq)rsrrpprogramZeach_varr3r3r4rusD      zDataFeeder.__init__c sg}t|j|j|jD]\}}}|t|j|||dq fddtj rQ|D]%}t |t |ks@Jdt |t |ft||D] \}}||qEq*n(|D]%}t |t |ksiJdt |t |ft||D] \}}| |qnqSi} t|j |D] \} }| | | <q| S)au According to :code:`feed_list` of :code:`DataFeeder` and :code:`iterable` , converts the input into a data structure that can feed into Executor. Parameters: iterable (generator): user defined python generator to read the raw input data Returns: :code:`dict`: a :code:`dict` that contains (variable name - converted tensor) pairs Example: .. code-block:: python >>> # In this example, reader - generator will return a list of ndarray of 3 elements >>> # feed API will convert each ndarray input into a tensor >>> # the return result is a dict with keys: data_1, data_2, data_3 >>> # result['data_1'] a LoD-Tensor with shape of [5, 2, 1, 3]. 5 is batch size, and [2, 1, 3] is the real shape of data_1. >>> # result['data_2'], result['data_3'] are similar. >>> import numpy as np >>> import paddle >>> from paddle import base >>> paddle.enable_static() >>> def reader(limit=5): ... for i in range(1, limit + 1): ... yield np.ones([6]).astype('float32') * i , np.ones([1]).astype('int64') * i, np.random.random([9]).astype('float32') ... >>> data_1 = paddle.static.data(name='data_1', shape=[None, 2, 1, 3]) >>> data_2 = paddle.static.data(name='data_2', shape=[None, 1], dtype='int64') >>> data_3 = paddle.static.data(name='data_3', shape=[None, 3, 3], dtype='float32') >>> feeder = base.DataFeeder(['data_1','data_2', 'data_3'], paddle.CPUPlace()) >>> result = feeder.feed(reader()) >>> print(result['data_1']) >>> print(result['data_2']) >>> print(result['data_3']) rcs4t|ttfr|D]}||q dS||dSr|)rElistrbr~) converterr?rl feed_datar3r4rs  z"DataFeeder.feed..feed_datazDThe number of fields in data (%d) does not match len(feed_list) (%d))rrrrrrnrppaddle frameworkZ use_pir_apirr~rr) rsiterablerrqr>rCrrrZret_dictZ each_namer3rr4r~sF(     zDataFeeder.feedcCs.|durt|St|jtjrttStSr|)intrErpr Z CUDAPlacerrr)rsZ num_placesr3r3r4_get_number_of_places_&s  z!DataFeeder._get_number_of_places_r|)rJrr__doc__rur~rr3r3r3r4rps  ?# Sr)r()rCrrDr)rO) r>rrYrKrerrfrrgrhrDri)E __future__rr.typingrrrnumpyr:rrrZpir.corer r rOr rr rrrrrrZtyping_extensionsrZpaddle._typingrrZpaddle._typing.dtype_likerrIrbr__annotations____all__rFrGZBOOLZ FP8_E4M3FNZFP8_E5M2ZFP16ZBF16ZFP32ZFP64ZINT8ZINT16ZINT32ZINT64ZUINT8Z COMPLEX64Z COMPLEX128STRINGZRAWrHZ_NUMPY_DTYPE_2_PADDLE_DTYPErArBrNr\rTrUrrrmrnrrr3r3r3r4s   $                    < "  +7(