o 1 i8N@sddlmZddlmZddlmZmZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZmZdd lmZmZmZmZe\ZZZe\ZZd Zd Zd Z d Z!eeddddKde"de"dej#fddZ$eeddd dLdeej#dee%dej#fddZ&edMdede%defddZ'e  dNdej#d ej#d!eej#d"ee(dej#f d#d$Z)e   dOd%ed&eed'e%d(e%def d)d*Z*ed+d,Z+edPdej#d.e,dej#fd/d0Z-edej#dej#fd1d2Z.e    -dQd ej#d!eej#d3eej#de%d4e,f d5d6Z/edd-d7ej0fdeee"fd8e"d9e,d:e,d;e1dej#f dd?Z3edRdej#d@e,dej#fdAdBZ4edSdej#dCe%dej#fdDdEZ5e dTdeej#e6fdGe"dHee,dej#fdIdJZ7dS)U) OrderedDict)MappingProxyType)ListOptionalN)Discrete MultiDiscrete) Deprecated) PublicAPI) try_import_tftry_import_torch) SpaceStructTensorStructType TensorTypeUniongư>iz)RLlib itself has no use for this anymore.F)helperror@sizealignreturncCs||j}tj||dtjd}|jj|}|dkrdn||}|dkr5|||ddd|}n |||||}t||ksLJt||jj|dksZJ|jj|S)aReturns an array of a given size that is 64-byte aligned. The returned array can be efficiently copied into GPU memory by TensorFlow. Args: size: The size (total number of items) of the array. For example, array([[0.0, 1.0], [2.0, 3.0]]) would have size=4. dtype: The numpy dtype of the array. align: The alignment to use. Returns: A np.ndarray with the given specifications. dtyper)itemsizenpemptyuint8ctypesdataviewlen)rrrnrZ data_alignoffsetoutputr&a/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/rllib/utils/numpy.py aligned_arrays   r(items time_majorcCsvt|dkrgSt|dkr|dSt|dtjr|djtjtjtjfvr|dj}tt dd|D|}|durx|dur\t dd|D}|dj d|f|dj dd}n1t d d|D}||dj df|dj dd}nt d d|D}|f|dj dd}| |}|j j d dksJ|j j tj|||rdndd |Stj||rdd Sdd S)a Concatenate arrays, ensuring the output is 64-byte aligned. We only align float arrays; other arrays are concatenated as normal. This should be used instead of np.concatenate() to improve performance when the output array is likely to be fed into TensorFlow. Args: items: The list of items to concatenate and align. time_major: Whether the data in items is time-major, in which case, we will concatenate along axis=1. Returns: The concat'd and aligned array. rrcss|]}|jVqdSN)r.0sr&r&r' asz!concat_aligned..NTcs|]}|jdVqdS)rNshaper,r&r&r'r/drcsr0rNr1r,r&r&r'r/ir3csr0r4r1r,r&r&r'r/nr3r)outaxisr6)r" isinstancerndarrayrfloat32Zfloat64rr(sumr2reshaperr concatenate)r)r*rZflatZ batch_dimZ new_shaper%r&r&r'concat_aligned=s@    r>Tx reduce_typecsfdd}t||S)aConverts values in `stats` to non-Tensor numpy or python types. Args: x: Any (possibly nested) struct, the values in which will be converted and returned as a new struct with all torch/tf tensors being converted to numpy types. reduce_type: Whether to automatically reduce all float64 and int64 data into float32 and int32 data, respectively. Returns: A new struct with the same structure as `x`, but with all values converted to numpy arrays (on CPU). cstrt|tjrt|dkr|n|}nt r:t|t jt j fr:t |dr:t s5J|}n|}rat|t jrat |jt jrT|t j}|St |jtra|t j}|S)Nrnumpy)torchr8Tensorr"rcpuitemdetachrAtfVariablehasattrexecuting_eagerlyrr9Z issubdtyperZfloatingastyper:intint32)rEretr@r&r'mappings*     z!convert_to_numpy..mapping)treeZ map_structure)r?r@rPr&rOr'convert_to_numpyxs  rRweightsbiases frameworkcCstd dd}||}|dko |jd|jdko |jd|jdk}|||d}||}t|||dur7d S|S) aCalculates FC (dense) layer outputs given weights/biases and input. Args: x: The input to the dense layer. weights: The weights matrix. biases: The biases vector. All 0s if None. framework: An optional framework hint (to figure out, e.g. whether to transpose torch weight matrices). Returns: The dense layer's output. FcSsRtrt|tjr|}tr tr t|tjr |}|r't |}|Sr+) rBr8rCrDrFrArGrJrHr transpose)r rVr&r&r'map_s    zfc..map_rBrr)rVNF)r2rmatmul)r?rSrTrUrWrVr&r&r'fcs  & r[inputs spaces_struct time_axis batch_axisc s|r|sJt|}|durt|ndgt|}d}d}g}t||D]\} |dur<|r<jd}|r<jd}t| tr[|rLt||g| t | j d tj q&t| tr|rlt||dg|r| tjfddt| jDddq&| tjfd dt| jDddq&ttrtg|rt||dgn|rt|dgntdg| tj q&tj|dd} |rt| ||dg} | S) a Flattens arbitrary input structs according to the given spaces struct. Returns a single 1D tensor resulting from the different input components' values. Thereby: - Boxes (any shape) get flattened to (B, [T]?, -1). Note that image boxes are not treated differently from other types of Boxes and get flattened as well. - Discrete (int) values are one-hot'd, e.g. a batch of [1, 0, 3] (B=3 with Discrete(4) space) results in [[0, 1, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]. - MultiDiscrete values are multi-one-hot'd, e.g. a batch of [[0, 2], [1, 4]] (B=2 with MultiDiscrete([2, 5]) space) results in [[1, 0, 0, 0, 1, 0, 0], [0, 1, 0, 0, 0, 0, 1]]. Args: inputs: The inputs to be flattened. spaces_struct: The (possibly nested) structure of the spaces that `inputs` belongs to. time_axis: Whether all inputs have a time-axis (after the batch axis). If True, will keep not only the batch axis (0th), but the time axis (1st) as-is and flatten everything from the 2nd axis up. batch_axis: Whether all inputs have a batch axis. If True, will keep that batch axis as-is and flatten everything from the other dims up. Returns: A single 1D tensor resulting from concatenating all flattened/one-hot'd input components. Depending on the time_axis flag, the shape is (B, n) or (B, T, n). .. testcode:: :skipif: True # B=2 from ray.rllib.utils.tf_utils import flatten_inputs_to_1d_tensor from gymnasium.spaces import Discrete, Box out = flatten_inputs_to_1d_tensor( {"a": [1, 0], "b": [[[0.0], [0.1]], [1.0], [1.1]]}, spaces_struct=dict(a=Discrete(2), b=Box(shape=(2, 1))) ) print(out) # B=2; T=2 out = flatten_inputs_to_1d_tensor( ([[1, 0], [0, 1]], [[[0.0, 0.1], [1.0, 1.1]], [[2.0, 2.1], [3.0, 3.1]]]), spaces_struct=tuple([Discrete(2), Box(shape=(2, ))]), time_axis=True ) print(out) .. testoutput:: [[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]] # B=2 n=4 [[[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]], [[1.0, 0.0, 2.0, 2.1], [0.0, 1.0, 3.0, 3.1]]] # B=2 T=2 n=4 Nrrdepthcs0g|]\}}tdd|f|dtjqS)Nr`one_hotrKrr:r-ir#Zinput_r&r' -sz/flatten_inputs_to_1d_tensor..r7cs(g|]\}}t||dtjqS)r`rcrergr&r'rh7s)rQflattenr"zipr2r8rrr<appendrdr#rKr:rr= enumerateZnvecfloatarray) r\r]r^r_Z flat_inputsZ flat_spacesBTr5spacemergedr&rgr'flatten_inputs_to_1d_tensorsd B           rscCsHt|tjr|jdd|St|trtt|St|tr"t|S|S)a{Flags actions immutable to notify users when trying to change them. Can also be used with any tree-like structure containing either dictionaries, numpy arrays or already immutable objects per se. Note, however that `tree.map_structure()` will in general not include the shallow object containing all others and therefore immutability will hold only for all objects contained in it. Use `tree.traverse(fun, action, top_down=False)` to include also the containing object. Args: obj: The object to be made immutable. Returns: The immutable object. .. testcode:: :skipif: True import tree import numpy as np from ray.rllib.utils.numpy import make_action_immutable arr = np.arange(1,10) d = dict(a = 1, b = (arr, arr)) tree.traverse(make_action_immutable, d, top_down=False) F)write)r8rr9Zsetflagsrrdict)objr&r&r'make_action_immutableSs     rw?deltacCs6tt||kt|dd|t|d|S)z4Reference: https://en.wikipedia.org/wiki/Huber_loss.@g?)rwhereabspower)r?ryr&r&r' huber_losszs.r~cCstt|dS)zComputes half the L2 norm of a tensor (w/o the sqrt): sum(x**2) / 2. Args: x: The input tensor. Returns: The l2-loss output according to the above formula given `x`. rz)rr;Zsquare)r?r&r&r'l2_losss rinitial_internal_states forget_biascCs|j|rdnd}|j|rdnd}|jdd}|dur.tj||fd} tj||fd} n|d} |d} |rBtj|||fd} n tj|||fd} t|D]} |r^|| ddddfn |dd| ddf} tj| | fdd} t| ||}t|dd|d|df|}t| |} t|ddd|f}t|dd||df}t | t||} t|dd|d|df}t|t| } |r| | | ddddf<qO| | dd| ddf<qO| | | ffS) aCalculates LSTM layer output given weights/biases, states, and input. Args: x: The inputs to the LSTM layer including time-rank (0th if time-major, else 1st) and the batch-rank (1st if time-major, else 0th). weights: The weights matrix. biases: The biases vector. All 0s if None. initial_internal_states: The initial internal states to pass into the layer. All 0s if None. time_major: Whether to use time-major or not. Default: False. forget_bias: Gets added to first sigmoid (forget gate) output. Default: 1.0. Returns: Tuple consisting of 1) The LSTM layer's output and 2) Tuple: Last (c-state, h-state). rrNr1r7r) r2rZzerosranger=rZsigmoidmultiplytanhadd)r?rSrTrr*rZsequence_lengthZ batch_sizeZunitsZc_statesZh_statesZunrolled_outputstZ input_matrixZinput_matmul_matrixZ sigmoid_1Z sigmoid_2Ztanh_3Z sigmoid_4r&r&r'lstms4 0$   rrXraon_value off_valuerc CsDt|trtj|tjd}n trt|tjr|}|jtj kr(| tj }d}|dkr3t |d}t ||ksEJd t |||j}tjg||Rd|}g}t|jD]5}dg|j} dg|j} d| |<t||| } |dkr||d| |d<t| | } || q\||||t|<| |S)aOne-hot utility function for numpy. Thanks to qianyizhang: https://gist.github.com/qianyizhang/07ee1c15cad08afb03f5de69349efc30. Args: x: The input to be one-hot encoded. depth: The max. number to be one-hot encoded (size of last rank). on_value: The value to use for on. Default: 1.0. off_value: The value to use for off. Default: 0.0. Returns: The one-hot encoded equivalent of the input array. rrrrz.rbr7)rBr8rCrAr2rr=rl)r?Zdepthsr&rr'one_hot_multidiscretes ralphacCst||||S)zImplementation of the leaky ReLU function. y = x * alpha if x < 0 else x Args: x: The input values. alpha: A scaling ("leak") factor to use for negative x. Returns: The leaky ReLU output for x. )rmaximum)r?rr&r&r'relu"s r derivativecCs$|r|d|Sddt| S)aY Returns the sigmoid function applied to x. Alternatively, can return the derivative or the sigmoid function. Args: x: The input to the sigmoid function. derivative: Whether to return the derivative or not. Default: False. Returns: The sigmoid function (or its derivative) applied to x. r)rexp)r?rr&r&r'r2s rrbr6epsiloncCs.|pt}t|}t|tj||dd|S)a{Returns the softmax values for x. The exact formula used is: S(xi) = e^xi / SUMj(e^xj), where j goes over all elements in x. Args: x: The input to the softmax function. axis: The axis along which to softmax. epsilon: Optional epsilon as a minimum value. If None, use `SMALL_NUMBER`. Returns: The softmax over x. T)Zkeepdims) SMALL_NUMBERrrrr;)r?r6rZx_expr&r&r'softmaxFs r)rr+)T)NN)NFT)rx)NNFrx)rXrY)rbN)8 collectionsrtypesrtypingrrrArrQZgymnasium.spacesrrZray._common.deprecationrZray.rllib.utils.annotationsr Zray.rllib.utils.frameworkr r Zray.rllib.utils.typingr r rrZtf1rGZtfvrB_rZ LARGE_INTEGERZMIN_LOG_NN_OUTPUTZMAX_LOG_NN_OUTPUTrLr9r(boolr>rRstrr[rsrwrmr~rrr:typerdrrrlistrr&r&r&r's     6) )  &  E  <