o 1 i+@sddlZddlmZddlmZddlmZddlmZm Z m Z m Z ddl Z ddlmZmZmZddlmZddlmZdd lmZdd lmZmZdd lmZdd lm Z er^dd l!m"Z"e#e$Z%eGdddeZeGdddeZee ddGdddZ&dS)N) dataclass)cached_property)Path) TYPE_CHECKINGListOptionalUnion)CheckpointConfig FailureConfig ScalingConfig) RuntimeEnv) _DEPRECATED)StorageContext)FAIL_FAST_DEPRECATION_MESSAGE%TRAINER_RESOURCES_DEPRECATION_MESSAGE)date_str) PublicAPI) UserCallbackcszeZdZUdZdZeeed<dZe e ed<dZ ee ed<fddZ efd d Zed d Zed dZZS)r aJ Configuration for scaling training. Args: num_workers: The number of workers (Ray actors) to launch. Each worker will reserve 1 CPU by default. The number of CPUs reserved by each worker can be overridden with the ``resources_per_worker`` argument. If the number of workers is 0, the training function will run in local mode, meaning the training function runs in the same process. use_gpu: If True, training will be done on GPUs (1 per worker). Defaults to False. The number of GPUs reserved by each worker can be overridden with the ``resources_per_worker`` argument. resources_per_worker: If specified, the resources defined in this Dict is reserved for each worker. Define the ``"CPU"`` and ``"GPU"`` keys (case-sensitive) to override the number of CPU or GPUs used by each worker. placement_strategy: The placement strategy to use for the placement group of the Ray actors. See :ref:`Placement Group Strategies ` for the possible options. accelerator_type: [Experimental] If specified, Ray Train will launch the training coordinator and workers on the nodes with the specified type of accelerators. See :ref:`the available accelerator types `. Ensure that your cluster has instances with the specified accelerator type or is able to autoscale to fulfill the request. This field is required when `use_tpu` is True and `num_workers` is greater than 1. use_tpu: [Experimental] If True, training will be done on TPUs (1 TPU VM per worker). Defaults to False. The number of TPUs reserved by each worker can be overridden with the ``resources_per_worker`` argument. This arg enables SPMD execution of the training workload. topology: [Experimental] If specified, Ray Train will launch the training coordinator and workers on nodes with the specified topology. Topology is auto-detected for TPUs and added as Ray node labels. This arg enables SPMD execution of the training workload. This field is required when `use_tpu` is True and `num_workers` is greater than 1. Ntrainer_resourcesFuse_tputopologycs|jdur tt|jr|jrtd|js|jdkrtd|jr+|jdkr+td|jrA|jdkrA|js:td|j sAtd|jdkrKt dt dS) Nz6Cannot specify both `use_gpu=True` and `use_tpu=True`.rz`use_tpu` is False but `TPU` was found in `resources_per_worker`. Either set `use_tpu` to True or remove `TPU` from `resources_per_worker.z`use_tpu` is True but `TPU` is set to 0 in `resources_per_worker`. Either set `use_tpu` to False or request a positive number of `TPU` in `resources_per_worker.zY`topology` must be specified in ScalingConfig when `use_tpu=True` and `num_workers` > 1.z``accelerator_type` must be specified in ScalingConfig when `use_tpu=True` and `num_workers` > 1.zRunning in local mode. The training function will run in the same process. If you are using it and running into issues please file a report at https://github.com/ray-project/ray/issues.)rDeprecationWarningrZuse_gpur ValueErrornum_tpus_per_workerZ num_workersrZaccelerator_typeloggerinfosuper __post_init__self __class__c/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/ray/train/v2/api/config.pyrJs4   zScalingConfig.__post_init__cs |jdur |jr ddiStjS)NTPUr)Zresources_per_workerrr_resources_per_worker_not_nonerr!r#r$r&us z,ScalingConfig._resources_per_worker_not_nonecCsiSNr#rr#r#r$_trainer_resources_not_none}sz)ScalingConfig._trainer_resources_not_nonecCs|jddS)z%The number of TPUs to set per worker.r%r)r&getrr#r#r$rsz!ScalingConfig.num_tpus_per_worker)__name__ __module__ __qualname____doc__rrdict__annotations__rrboolrstrrpropertyr&r(r __classcell__r#r#r!r$r s & + r c@s:eZdZUdZeZeeefe d<dZ e e d<ddZ dS)r ahConfiguration related to failure handling of each training run. Args: max_failures: Tries to recover a run from training worker errors at least this many times. Will recover from the latest checkpoint if present. Setting to -1 will lead to infinite recovery retries. Setting to 0 will disable retries. Defaults to 0. controller_failure_limit: [DeveloperAPI] The maximum number of controller failures to tolerate. Setting to -1 will lead to infinite controller retries. Setting to 0 will disable controller retries. Defaults to -1. fail_fastcontroller_failure_limitcCs|jtkr ttdSr')r4r rrrr#r#r$rs zFailureConfig.__post_init__N) r*r+r,r-r r4rr0r1r/r6intrr#r#r#r$r s   r Zstable)Z stabilityc@seZdZUdZdZeeed<dZeeed<dZ ee j j ed<dZ eeed<dZeeed<dZeeded <dZeeeefed <eZeed <eZeed <eZeed <eZeed<eZeed<ddZedefddZ dS) RunConfigaDRuntime configuration for training runs. Args: name: Name of the trial or experiment. If not provided, will be deduced from the Trainable. storage_path: [Beta] Path where all results and checkpoints are persisted. Can be a local directory or a destination on cloud storage. For multi-node training/tuning runs, this must be set to a shared storage location (e.g., S3, NFS). This defaults to the local ``~/ray_results`` directory. storage_filesystem: [Beta] A custom filesystem to use for storage. If this is provided, `storage_path` should be a path with its prefix stripped (e.g., `s3://bucket/path` -> `bucket/path`). failure_config: Failure mode configuration. checkpoint_config: Checkpointing configuration. callbacks: [DeveloperAPI] A list of callbacks that the Ray Train controller will invoke during training. worker_runtime_env: [DeveloperAPI] Runtime environment configuration for all Ray Train worker actors. Nname storage_pathstorage_filesystemfailure_configcheckpoint_configr callbacksworker_runtime_env sync_configverbosestopprogress_reporter log_to_filecsddlm}|jdur||_|jst|_|jst|_t|jtr(|j |_d}gd}|D]}t ||t kr@t | |q0|jsKdt|_|jpOg|_|jpUi|_ddlmtfdd|jDsmtd t|jts}td |jjd t|jtstd |jjd dS)Nr)DEFAULT_STORAGE_PATHaI`RunConfig({})` is deprecated. This configuration was a Ray Tune API that did not support Ray Train usage well, so we are dropping support going forward. If you heavily rely on these configurations, you can run Ray Train as a single Ray Tune trial. See this issue for more context: https://github.com/ray-project/ray/issues/49454)r@rArBrCrDzray_train_run-RayTrainCallbackc3s|]}t|VqdSr') isinstance).0cbrFr#r$ sz*RunConfig.__post_init__..zAll callbacks must be instances of `ray.train.UserCallback`. Passing in a Ray Tune callback is no longer supported. See this issue for more context: https://github.com/ray-project/ray/issues/49454z!Invalid `CheckpointConfig` type: z|. Use `ray.train.CheckpointConfig` instead. See this issue for more context: https://github.com/ray-project/ray/issues/49454zInvalid `FailureConfig` type: zy. Use `ray.train.FailureConfig` instead. See this issue for more context: https://github.com/ray-project/ray/issues/49454)Zray.train.constantsrEr:r<r r=r rHras_posixgetattrr rformatr9rr>r?Zray.train.v2.api.callbackrGallrr")r rEZrun_config_deprecation_messageZunsupported_paramsparamr#rFr$rsD          zRunConfig.__post_init__returncCst|j|j|jdS)N)r:Zexperiment_dir_namer;)rr:r9r;rr#r#r$storage_contexts zRunConfig.storage_context)!r*r+r,r-r9rr1r/r:r;pyarrowfsZ FileSystemr<r r=r r>rr?rr.r r r@rArBrCrDrrrrRr#r#r#r$r8s"      Er8)'logging dataclassesr functoolsrpathlibrtypingrrrrZ pyarrow.fsrSZray.air.configr r ZFailureConfigV1r ZScalingConfigV1Zray.runtime_envr Z ray.train.v2._internal.constantsr Z(ray.train.v2._internal.execution.storagerZ&ray.train.v2._internal.migration_utilsrrZray.train.v2._internal.utilrZray.util.annotationsrZ ray.trainr getLoggerr*rr8r#r#r#r$s.          h