o 1 iN3@sddlmZmZmZmZmZmZmZmZddl Z ddl m Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)e!\Z*Z+e \Z,Z+edddddddddddddddddddddde-de-deeeeeeegee(ee(ffdeege)fdeeeegee-e(ffdeeeeeeeefeegefdeeeee-e(fee(eegee-e(ffdeeede(gee-e(ffdeeegee-e(ffdeeee)gdfdeeee j.e j.e)gdfdeeee j.e j.e)gdfdeeee j/j.e j/j.e)gdfdeeee j.e j.e)gdfd eeee j/j.e j/j.e)gdfd!eee(ee(gee(e(ffd"eeeee(e(e(gee(e0ee(ffd#eeee j/j.e j/j.e)gefd$eeee j/j.e j/j.e)geeeeffd%eeeegee'e1ffd&eeedgdfd'eee0d(eeege2fd)eef0d*d+Z3dS),)AnyCallableDictListOptionalTupleTypeUnionN) ModelCatalog)ModelV2)TorchDistributionWrapper) TorchModelV2)Policy) SampleBatch) TorchPolicy) add_mixinsNullContextManager) OldAPIStackoverride)try_import_torchtry_import_jax)LEARNER_STATS_KEY)convert_to_numpy)ModelGradients TensorTypeAlgorithmConfigDict)get_default_configstats_fnpostprocess_fnextra_action_out_fnextra_grad_process_fnextra_learn_fetches_fn optimizer_fnvalidate_spaces before_initbefore_loss_init after_init_after_loss_initaction_sampler_fnaction_distribution_fn make_modelmake_model_and_action_distcompute_gradients_fnapply_gradients_fnmixinsget_batch_divisibility_reqname frameworkloss_fnrrrrr ztorch.optim.Optimizerr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/returncszttt|}G fddd|}fdd}t||_||_||_|S)aHelper function for creating a new Policy class at runtime. Supports frameworks JAX and PyTorch. Args: name: name of the policy (e.g., "PPOTorchPolicy") framework: Either "jax" or "torch". loss_fn (Optional[Callable[[Policy, ModelV2, Type[TorchDistributionWrapper], SampleBatch], Union[TensorType, List[TensorType]]]]): Callable that returns a loss tensor. get_default_config (Optional[Callable[[None], AlgorithmConfigDict]]): Optional callable that returns the default config to merge with any overrides. If None, uses only(!) the user-provided PartialAlgorithmConfigDict as dict for this Policy. postprocess_fn (Optional[Callable[[Policy, SampleBatch, Optional[Dict[Any, SampleBatch]], Optional[Any]], SampleBatch]]): Optional callable for post-processing experience batches (called after the super's `postprocess_trajectory` method). stats_fn (Optional[Callable[[Policy, SampleBatch], Dict[str, TensorType]]]): Optional callable that returns a dict of values given the policy and training batch. If None, will use `TorchPolicy.extra_grad_info()` instead. The stats dict is used for logging (e.g. in TensorBoard). extra_action_out_fn (Optional[Callable[[Policy, Dict[str, TensorType], List[TensorType], ModelV2, TorchDistributionWrapper]], Dict[str, TensorType]]]): Optional callable that returns a dict of extra values to include in experiences. If None, no extra computations will be performed. extra_grad_process_fn (Optional[Callable[[Policy, "torch.optim.Optimizer", TensorType], Dict[str, TensorType]]]): Optional callable that is called after gradients are computed and returns a processing info dict. If None, will call the `TorchPolicy.extra_grad_process()` method instead. # TODO: (sven) dissolve naming mismatch between "learn" and "compute.." extra_learn_fetches_fn (Optional[Callable[[Policy], Dict[str, TensorType]]]): Optional callable that returns a dict of extra tensors from the policy after loss evaluation. If None, will call the `TorchPolicy.extra_compute_grad_fetches()` method instead. optimizer_fn (Optional[Callable[[Policy, AlgorithmConfigDict], "torch.optim.Optimizer"]]): Optional callable that returns a torch optimizer given the policy and config. If None, will call the `TorchPolicy.optimizer()` method instead (which returns a torch Adam optimizer). validate_spaces (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable that takes the Policy, observation_space, action_space, and config to check for correctness. If None, no spaces checking will be done. before_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable to run at the beginning of `Policy.__init__` that takes the same arguments as the Policy constructor. If None, this step will be skipped. before_loss_init (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], None]]): Optional callable to run prior to loss init. If None, this step will be skipped. after_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): DEPRECATED: Use `before_loss_init` instead. _after_loss_init (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], None]]): Optional callable to run after the loss init. If None, this step will be skipped. This will be deprecated at some point and renamed into `after_init` to match `build_tf_policy()` behavior. action_sampler_fn (Optional[Callable[[TensorType, List[TensorType]], Tuple[TensorType, TensorType]]]): Optional callable returning a sampled action and its log-likelihood given some (obs and state) inputs. If None, will either use `action_distribution_fn` or compute actions by calling self.model, then sampling from the so parameterized action distribution. action_distribution_fn (Optional[Callable[[Policy, ModelV2, TensorType, TensorType, TensorType], Tuple[TensorType, Type[TorchDistributionWrapper], List[TensorType]]]]): A callable that takes the Policy, Model, the observation batch, an explore-flag, a timestep, and an is_training flag and returns a tuple of a) distribution inputs (parameters), b) a dist-class to generate an action distribution object from, and c) internal-state outputs (empty list if not applicable). If None, will either use `action_sampler_fn` or compute actions by calling self.model, then sampling from the parameterized action distribution. make_model (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], ModelV2]]): Optional callable that takes the same arguments as Policy.__init__ and returns a model instance. The distribution class will be determined automatically. Note: Only one of `make_model` or `make_model_and_action_dist` should be provided. If both are None, a default Model will be created. make_model_and_action_dist (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], Tuple[ModelV2, Type[TorchDistributionWrapper]]]]): Optional callable that takes the same arguments as Policy.__init__ and returns a tuple of model instance and torch action distribution class. Note: Only one of `make_model` or `make_model_and_action_dist` should be provided. If both are None, a default Model will be created. compute_gradients_fn (Optional[Callable[ [Policy, SampleBatch], Tuple[ModelGradients, dict]]]): Optional callable that the sampled batch an computes the gradients w.r. to the loss function. If None, will call the `TorchPolicy.compute_gradients()` method instead. apply_gradients_fn (Optional[Callable[[Policy, "torch.optim.Optimizer"], None]]): Optional callable that takes a grads list and applies these to the Model's parameters. If None, will call the `TorchPolicy.apply_gradients()` method instead. mixins (Optional[List[type]]): Optional list of any class mixins for the returned policy class. These mixins will be applied in order and will have higher precedence than the TorchPolicy class. get_batch_divisibility_req (Optional[Callable[[Policy], int]]): Optional callable that returns the divisibility requirement for sample batches. If None, will assume a value of 1. Returns: Type[TorchPolicy]: TorchPolicy child class constructed from the specified args. cseZdZ fddZee dfdd Ze fddZe fdd Zefd d Z efd d Z e fddZ efddZ efddZ ddZddZZS)z&build_policy_class..policy_clsc s||_|_|jd< r ||||jr||||j r? dus)Jd |||||_tj||jdd\}}n( rL ||||\|_}ntj||jdd\}}tj||||jdd|_t}t|j|ssJd |_|jj |||||j|jdrdn||ddd |j |jj p}|r|||j |j ||jd |jdrdn d r||||d |_dS) Nr1zAEither `make_model` or `make_model_and_action_dist` must be None!model)r1) obs_space action_spaceZ num_outputsZ model_configr1z5ERROR: Generated Model must be a TorchModelV2 object!Z in_evaluation max_seq_len) observation_spacer6configr4lossZaction_distribution_classr(r)r7r/T)Zauto_remove_unneeded_view_reqsrr)r9r1r4r Zget_action_distZ get_model_v2r isinstance parent_cls__init__Zview_requirementsupdater8r6Z!_initialize_loss_from_dummy_batchZglobal_timestep) selfr5r6r9Z dist_class_Z logit_dimZ model_clsZ_before_loss_init)r'r)r(r&r$r%r1r/r2r*r+r<rr#l/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/rllib/policy/policy_template.pyr=sz          z/build_policy_class..policy_cls.__init__Ncs`|"t|||}r||||WdS|WdS1s)wYdSN)_no_grad_contextsuperpostprocess_trajectory)r?Z sample_batchZother_agent_batchesZepisode) __class__rrArBrFLs  $z=build_policy_class..policy_cls.postprocess_trajectorycsr|||S|||S)zCalled after optimizer.zero_grad() and loss.backward() calls. Allows for gradient processing before optimizer.step() is called. E.g. for gradient clipping. )extra_grad_process)r? optimizerr:)r r<rArBrH_s z9build_policy_class..policy_cls.extra_grad_processcs.rt|}ttiifi|S|SrC)rdictrextra_compute_grad_fetches)r?Zfetches)r!r<rArBrKks  zAbuild_policy_class..policy_cls.extra_compute_grad_fetchescsr||S||SrC)compute_gradients)r?batch)r,r<rArBrLts  z8build_policy_class..policy_cls.compute_gradientscs"r ||dS||dSrC)apply_gradients)r?Z gradients)r-r<rArBrN{sz6build_policy_class..policy_cls.apply_gradientscs^|!r|||||}n |||||}||WdS1s(wYdSrC)rDextra_action_out_convert_to_numpy)r?Z input_dictZ state_batchesr4Z action_dist stats_dict)rr<rArBrOs   $z7build_policy_class..policy_cls.extra_action_outcs"r ||j}|S|}|SrC)r9rI)r?Z optimizers)r"r<rArBrIs   z0build_policy_class..policy_cls.optimizercsT|r ||}n|j||}||WdS1s#wYdSrC)rDr<extra_grad_inforP)r?Z train_batchrQ)rrArBrRs  $z6build_policy_class..policy_cls.extra_grad_infocSs|jdkr tStSNtorch)r1rTZno_gradr)r?rArArBrDs z7build_policy_class..policy_cls._no_grad_contextcSs|jdkr t|S|SrS)r1r)r?datarArArBrPs z8build_policy_class..policy_cls._convert_to_numpy)NN)__name__ __module__ __qualname__r=rrrFrHrKrLrNrOrIrRrDrP __classcell__rA)r'r)r(r&r-r$r%r,rr r!r1r/r2r*r+r"r<rrr#)rGrB policy_clss*&U  rZcstditfi|S)aCreates a Torch|JAXPolicy cls based on settings of another one. Keyword Args: **overrides: The settings (passed into `build_torch_policy`) that should be different from the class that this method is called on. Returns: type: A new Torch|JAXPolicy sub-class. Examples: >> MySpecialDQNPolicyClass = DQNTorchPolicy.with_updates( .. name="MySpecialDQNPolicyClass", .. loss_function=[some_new_loss_function], .. ) NrA)build_policy_classrJ)Z overrides)original_kwargsrArB with_updatessz(build_policy_class..with_updates)localscopyrr staticmethodr]rVrX)r0r1r2rrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/baserZr]rA)r'r)r(r&r-r$r%r,rr r!r1r/r2r*r+r"r\r<rrr#rBr[ s S < 5 r[)4typingrrrrrrrr Z gymnasiumZgymZray.rllib.models.catalogr Zray.rllib.models.modelv2r Z(ray.rllib.models.torch.torch_action_distr Z$ray.rllib.models.torch.torch_modelv2r Zray.rllib.policy.policyrZray.rllib.policy.sample_batchrZray.rllib.policy.torch_policyrZray.rllib.utilsrrZray.rllib.utils.annotationsrrZray.rllib.utils.frameworkrrZ$ray.rllib.utils.metrics.learner_inforZray.rllib.utils.numpyrZray.rllib.utils.typingrrrZjaxr@rTstrZSpacespacestyperJintr[rArArArBs4(                   #'(+.169>A GLRU XYZ