o )i +@sddlZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl m Z dee fddZdZzdd lmZmZmZdd lmZed ZeZd ZWneycdZdZdZdZdZdZYnweeeeefZejGd ddZdeddfddZdeddfddZdeegefdeeegdfde j j!j"fddZ#edeegefdeeegdfddfddZ$GdddZ%dS)N)contextmanager)AnyCallableOptionalUnion)is_pin_memory_availablereturncCsd}td}|D] }||vr|}nq Wdn1swY|dur)dS|d}||d}|dd}|dd|sQJd|d||S) a) According to according to https://man7.org/linux/man-pages/man5/proc_pid_maps.5.html, the file `/proc/self/maps` contains the memory maps of the process, which includes the shared libraries loaded by the process. We can use this file to find the path of the a loaded library. Nz/proc/self/maps/z.sorzUnexpected filename: z for library )openindexstripsplit rpartition startswith)lib_nameZ found_lineflinestartpathfilenamerg/home/app/PaddleOCR-VL-test/.venv_paddleocr/lib/python3.10/site-packages/vllm/device_allocator/cumem.pyfind_loaded_librarys$  rF) init_modulepython_create_and_mappython_unmap_and_release) CudaRTLibraryZcumem_allocatorTc@s0eZdZUeed<eed<dZeej ed<dS)AllocationDatahandletagNcpu_backup_tensor) __name__ __module__ __qualname__ HandleType__annotations__strr!rtorchZTensorrrrrrFs rallocation_handlecC t|dSN)rr)rrrcreate_and_mapM r-cCr*r+)rr,rrrunmap_and_releaseQr.r/python_malloc_fnpython_free_funccCs t||tjjtdd}|S)NZ my_mallocZmy_free)rr(cudamemoryCUDAPluggableAllocatorr)r0r1 new_allocrrrget_pluggable_allocatorUs r6ccsZt||}tjj|j}tjj|||fVWddS1s&wYdSr+)r6r(r2r3ZMemPoolZ _allocatorZ use_mem_pool)r0r1r5Zmem_poolrrruse_memory_pool_with_allocator`s   "r7c@seZdZUdZdZded<dZeed<edddZ d d Z d e ddfd d Z de de fddZ ddeeeedfefddfddZddeeeddfddZeddeefddZde fddZdS)CuMemAllocatora' A singleton class that manages a memory pool for CUDA tensors. The memory in this pool can be offloaded or discarded when the allocator sleeps. Inside the `use_memory_pool(tag)` context, all tensors created will be allocated in the memory pool, and has the same tag as the tag passed to the context. When we call `sleep`, all tensors with the specified tag will be offloaded to CPU memory, and the rest of the tensors will be discarded. When we call `wake_up`, all tensors that are previously offloaded will be loaded back to GPU memory, and the rest of the tensors will have empty memory. Why it needs to be a singleton? When allocated tensors are garbage collected, PyTorch will call the free callback, which will call the `python_free_callback` method. The C-extension uses a global variable to store the function of an instance of this class. If we create multiple instances of this class, the global variable will be overwritten and the free callback will not work as expected. Ninstancedefault default_tagrcCs$tsJdtjdurtt_tjS)z CuMemAllocator is a singleton class. We cannot call the constructor directly. Call this method to get the instance. z cumem allocator is not availableN)cumem_availabler8r9rrrr get_instances  zCuMemAllocator.get_instancecCs6tjdd}d|vsJdi|_tj|_i|_dS)NZPYTORCH_CUDA_ALLOC_CONFzexpandable_segments:TruezExpandable segments are not compatible with memory pool. Please track https://github.com/pytorch/pytorch/issues/147851 for the latest updates.)osenvirongetpointer_to_datar8r; current_tagallocator_and_pools)selfconfrrr__init__s  zCuMemAllocator.__init__r)cCs|d}t||j|j|<dS)zj Internal method to store the allocation data when memory is allocated in the memory pool.N)rrCrB)rEr)Zpy_d_memrrrpython_malloc_callbacks  z%CuMemAllocator.python_malloc_callbackptrcCs"|j|}|jdurd|_|jS)zh Internal method to look up the allocation data when memory is freed in the memory pool.N)rBpopr!r)rErJdatarrrpython_free_callbacks  z#CuMemAllocator.python_free_callback offload_tags.cCs|dur tjf}nt|tr|f}t|tsJ|jD]-\}}|j}|j|vrF|d}t j |t j dt d}| }t|||||_t|qtt jdS)aS Put the allocator in sleep mode. All data in the memory allocation with the specified tag will be offloaded to CPU memory, and others will be discarded. :param offload_tags: The tags of the memory allocation that will be offloaded. The rest of the memory allocation will be discarded. Ncpu)ZdtypeZdeviceZ pin_memory)r8r; isinstancer'tuplerBitemsrr r(emptyZuint8rdata_ptr libcudart cudaMemcpyr!r/gcZcollectr2Z empty_cache)rErNrJrLr size_in_bytesr!cpu_ptrrrrsleeps*    zCuMemAllocator.sleeptagscCs||jD]6\}}|dus|j|vr;|j}t||jdur;|j}|dur;||}|}t |||d|_qdS)a Wake up the allocator from sleep mode. All data that is previously offloaded will be loaded back to GPU memory, and the rest of the data will have empty memory. :param tags: The tags of the memory allocation that will be loaded back to GPU memory. If None, all memory allocation will be loaded back to GPU memory. N) rBrSr rr-r!ZnumelZ element_sizerUrVrW)rEr\rJrLrr!rYrZrrrwake_ups  zCuMemAllocator.wake_upr ccst|durtj}t|tsJ|j}||_t|j|j}||j|<dV||_WddS1s3wYdS)a' A context manager to use the memory pool. All memory allocation created inside the context will be allocated in the memory pool, and has the specified tag. :param tag: The tag of the memory allocation. If None, the default tag will be used. N) r8r;rQr'rCr7rIrMrD)rEr Zold_tagrLrrruse_memory_pools   "zCuMemAllocator.use_memory_poolcCs.d}|jD] \}}|j}||d7}q|S)zM Get the total number of bytes allocated in the memory pool. rrO)rBrSr)rEZ sum_bytesrJrLrrrrget_current_usages z CuMemAllocator.get_current_usage)rr8r+)r"r#r$__doc__r9r&r;r' staticmethodr=rGr%rIintrMrrrRr[listr]rr^r_rrrrr8js*         &%r8)& dataclassesrXr? contextlibrtypingrrrrr(Z vllm.utilsrr'rr<Zvllm.cumem_allocatorrrrZ2vllm.distributed.device_communicators.cuda_wrapperrrrVModuleNotFoundErrorrRrbr% dataclassrr-r/r2r3r4r6r7r8rrrrs`