# FBGEMM_GPU Installation Instructions The most up-to-date instructions are embedded in [`setup_env.bash`](../../.github/scripts/setup_env.bash). The general steps for building FBGEMM_GPU are as follows: 1. Set up an isolated environment for CUDA, ROCm, or CPU runtime 1. Install the relevant Python libraries 1. Install PyTorch 1. Install the FBGEMM_GPU package 1. Run post-installation checks The shortened summary of the installation steps: ```sh # CUDA Nightly pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121/ pip install --pre fbgemm-gpu --index-url https://download.pytorch.org/whl/nightly/cu121/ # CUDA Release pip install torch --index-url https://download.pytorch.org/whl/cu121/ pip install fbgemm-gpu --index-url https://download.pytorch.org/whl/cu121/ # CPU-only Nightly pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cpu/ pip install --pre fbgemm-gpu --index-url https://download.pytorch.org/whl/nightly/cpu/ # CPU-only Release pip install torch --index-url https://download.pytorch.org/whl/cpu/ pip install fbgemm-gpu --index-url https://download.pytorch.org/whl/cpu/ # Test the installation python -c "import torch; import fbgemm_gpu" ``` ## Set Up CUDA Environment The CUDA variant of FBGEMM_GPU requires an NVIDIA GPU installed to the machine, along with working NVIDIA drivers installed; otherwise or the library will fall back to running the CPU version of the operators. The FBGEMM_GPU CUDA package is currently only built for the SM70 and SM80 architectures (V100 and A100 GPUs respectively). Support for other architectures can be achieved by building the pacakge from scratch, but is not guaranteed to work. ### Install NVIDIA Drivers The NVIDIA display drivers must be installed on the system prior to all other environment setup. The steps provided by [NVIDIA](https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html) and [PyTorch](https://github.com/pytorch/test-infra/blob/main/.github/actions/setup-nvidia/action.yml) are the most authoritative instructions for doing this. Driver setup may be verified with the `nvidia-smi` command: ```sh nvidia-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.76 Driver Version: 515.76 CUDA Version: 11.7 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | | 0% 31C P0 59W / 300W | 0MiB / 23028MiB | 2% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ``` ### Set Up the Docker Container and Conda Environment It is recommended, though not required, to install and run FBGEMM_GPU through a Docker setup for isolation and reproducibility of the CUDA environment. The NVIDIA-Docker runtime needs to be installed to expose the driver to the container. The install steps provided by [PyTorch](https://github.com/pytorch/test-infra/blob/main/.github/actions/setup-nvidia/action.yml) provide details on how to achieve this. Once this is done, follow the steps in the [Build Instructions](./BuildInstructions.md) for pulling the CUDA Docker image and launching a container. From there, the rest of the runtime environment may be constructed through Conda. Follow the steps for installing MiniConda and setting up the Conda environment from the [Build Instructions](./BuildInstructions.md). ### Install CUDA If the OS / Docker environment does not already contain the full CUDA runtime, follow the steps in the [Build Instructions](./BuildInstructions.md) for installing the CUDA toolkit inside a Conda environment. ## Set Up ROCm Environment The ROCm variant of FBGEMM_GPU requires an AMD GPU installed to the machine, along with working AMDGPU drivers installed; otherwise or the library will fall back to running the CPU version of the operators. ### Install AMDGPU Drivers The AMDGPU display drivers must be installed on the system prior to all other environment setup. The steps provided by [AMD](https://docs.amd.com/bundle/ROCm-Installation-Guide-v5.5/page/How_to_Install_ROCm.html) are the most authoritative instructions for doing this. Driver setup may be verified with the `rocm-smi` command: ```sh rocm-smi ======================= ROCm System Management Interface ======================= ================================= Concise Info ================================= GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 0 33.0c 37.0W 300Mhz 1200Mhz 0% auto 290.0W 0% 0% 1 32.0c 39.0W 300Mhz 1200Mhz 0% auto 290.0W 0% 0% 2 33.0c 37.0W 300Mhz 1200Mhz 0% auto 290.0W 0% 0% ================================================================================ ============================= End of ROCm SMI Log ============================== ``` ### Set Up the Docker Container and Conda Environment It is recommended, though not required, to install and run FBGEMM_GPU through a Docker setup for isolation and reproducibility of the ROCm environment, which can be difficult to set up. Follow the steps in the [Build Instructions](./BuildInstructions.md) for pulling the full ROCm Docker image and launching a container. From there, the rest of the runtime environment may be constructed through Conda. Follow the steps for installing MiniConda and setting up the Conda environment from the [Build Instructions](./BuildInstructions.md). ## Install Python Libraries Install the other relevant Python libraries for working with FBGEMM_GPU: ```sh conda install -n "${env_name}" -y \ hypothesis \ numpy \ scikit-build ``` ## Install PyTorch Follow the steps in the [Build Instructions](./BuildInstructions.md) for installing PyTorch inside a Conda environment. ## Install the FBGEMM_GPU Package FBGEMM_GPU installation is done through PIP. ### Install the CUDA Variant ```sh # !! Run inside the Conda environment !! # Release GPU pip install fbgemm-gpu # Nightly GPU pip install fbgemm-gpu-nightly ``` ### Install the ROCm Variant As of time of writing, there no packages yet for the ROCm variant of FBGEMM_GPU. ### Install the CPU-Only Variant ```sh # !! Run inside the Conda environment !! # Release CPU pip install fbgemm-gpu-cpu # Nightly CPU pip install fbgemm-gpu-nightly-cpu ``` ## Post-Installation Checks After installation, run an import test to ensure that the library is correctly linked and set up. ```sh # !! Run inside the Conda environment !! python -c "import torch; import fbgemm_gpu; print(torch.ops.fbgemm.merge_pooled_embeddings)" ``` ### Undefined Symbols A common error that is encountered is the failure to import FBGEMM_GPU in Python, which has the following error signature: ```sh Traceback (most recent call last): File "/root/miniconda/envs/mycondaenv/lib/python3.10/site-packages/torch/_ops.py", line 565, in __getattr__ op, overload_names = torch._C._jit_get_operation(qualified_op_name) RuntimeError: No such operator fbgemm::jagged_2d_to_dense The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "/root/miniconda/envs/mycondaenv/lib/python3.10/site-packages/fbgemm_gpu-0.4.1.post47-py3.10-linux-aarch64.egg/fbgemm_gpu/__init__.py", line 21, in from . import _fbgemm_gpu_docs # noqa: F401, E402 File "/root/miniconda/envs/mycondaenv/lib/python3.10/site-packages/fbgemm_gpu-0.4.1.post47-py3.10-linux-aarch64.egg/fbgemm_gpu/_fbgemm_gpu_docs.py", line 18, in torch.ops.fbgemm.jagged_2d_to_dense, File "/root/miniconda/envs/mycondaenv/lib/python3.10/site-packages/torch/_ops.py", line 569, in __getattr__ raise AttributeError( AttributeError: '_OpNamespace' 'fbgemm' object has no attribute 'jagged_2d_to_dense' ERROR conda.cli.main_run:execute(47): `conda run python -c import fbgemm_gpu` failed. (See above for error) /root/miniconda/envs/mycondaenv/lib/python3.10/site-packages/fbgemm_gpu-0.4.1.post47-py3.10-linux-aarch64.egg/fbgemm_gpu/fbgemm_gpu_py.so: undefined symbol: _ZN6fbgemm48FloatOrHalfToFusedNBitRowwiseQuantizedSBHalfAvx2ItLi2EEEvPKT_miPh ``` Undefined symbols can appear in an FBGEMM_GPU installation for the following reasons: 1. The runtime libraries that FBGEMM_GPU depends on, such as `libnvidia-ml.so` or `libtorch.so`, are either not installed correctly or are not visible in `LD_LIBRARY_PATH`. 1. The FBGEMM_GPU package was built incorrectly and contains declarations that were not linked (see [1618](https://github.com/pytorch/FBGEMM/issues/1618) for example). In the former case, this may be resolved by re-installing the relevant packages and/or manually updating `LD_LIBRARY_PATH`. In the latter case, this is a serious building and packaging issue that should be reported to the FBGEMM developers.