ROCm packages in [community]

[tpkessler]

HIP in [community]

I'm excited to tell you that today, initial support for ROCm in the official Arch Linux repositories has been released!

What's included?

• Everything you need to render in blender with the new HIP backend;
• OpenCL runtime for darktable;
• system information with rocminfo and rocm-smi.

Where do I find it?

• Enable the testing repositories, including [community-testing].

WARNING: This may break your system!

pacman -Syyu && pacman -S hip-runtime-amd

• Add your login user to the video and render groups.
• [Optional] Add /opt/rocm/bin to your path.

How do I check if ROCm is correctly installed?

• rocminfo collects information on your AMD GPU.
• To test HIP you can render a (simple) scene in blender or you can compile and run the following simple test

/opt/rocm/bin/hipcc test.cpp -o test && ./test

Source code

#include <iostream>
#include <cmath>
#include <vector>
#include <hip/hip_runtime.h>

__global__
void saxpy(int n, float a, const float *x, float *y)
{
    int i = hipThreadIdx_x + hipBlockDim_x * hipBlockIdx_x;
    if(i < n){
        y[i] = a * x[i] + y[i];
    }
}

__global__
void sset(int n, float a, float *x)
{
    int i = hipThreadIdx_x + hipBlockDim_x * hipBlockIdx_x;
    if(i < n){
        x[i] = a;
    }
}

int main()
{
    hipDeviceProp_t prop;
    hipGetDeviceProperties(&prop, 0);
    std::cout << "Agent " << prop.name << "\n";
    std::cout << "System version " << prop.major
        << "." << prop.minor << "\n";

    int n = 1024;
    float *x;
    float *y;
    hipMalloc((void**)&x, sizeof *x * n);
    hipMalloc((void**)&y, sizeof *y * n);

    std::vector<float> xin(n);
    for(int i = 0; i < n; i++){
        xin[i] = -1.0 + 2.0 * i / n;
    }
    hipMemcpy(x, xin.data(), sizeof *x * n, hipMemcpyHostToDevice);

    float ac = -14.412f;
    hipLaunchKernelGGL(sset, dim3(1), dim3(n), 0, 0, n, ac, y);

    float a = 5321.124f;
    hipLaunchKernelGGL(saxpy, dim3(1), dim3(n), 0, 0, n, a, x, y);

    std::vector<float> yout(n);
    hipMemcpy(yout.data(), y, sizeof *y * n, hipMemcpyDeviceToHost);

    hipFree(x);
    hipFree(y);

    for(int i = 0; i < n; i++){
        yout[i] -= a * xin[i];
        if(std::abs(yout[i] - ac) > 0.001f){
            std::cout << "Test failed at index " << i
                << " with entry " << yout[i]
                << " (" << ac << ")\n";
            return 1;
        }
    }
    std::cout << "TESTS PASSED!" << std::endl;
}

Example output for a Vega 56:

Agent AMD Radeon RX Vega
System version 9.0
TESTS PASSED!

I want to do Machine Learning in X!

The ROCm stack will be released step by step. More and more packages will be added to [community] if the basic packages are sufficiently tested. As ML libraries rely on almost all ROCm packages, this will be the last step.

Feedback

Post your experiences with the new ROCm packages below!

[jellersby]

Relating my experience so far using these packages. Thank you for putting them together! Ultimately, my goal is to build and use rocm-arch/python-pytorch-rocm.

I'm using all available packages in community-testing, and supplementing those with the remaining rocm-arch PKGBUILDs: rocm-core, rocm-dbgapi, rocm-gdb, rocm-debug-agent, rocprofiler, and roctracer.

Additionally, I had to install openmp-extras from arch4edu because makepkg fails to build it from the rocm-arch PKGBUILD.

I'm able to build rocm-arch/python-pytorch-rocm successfully! However, when loading the torch module and calling torch.cuda.is_available(), I get the following:

>>> import torch
>>> torch.cuda.is_available()
/usr/include/c++/12.2.0/bits/stl_vector.h:1123: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = const char*; _Alloc = std::allocator<const char*>; reference = const char*&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
zsh: IOT instruction (core dumped)  python

Here's a truncated backtrace from gdb:

#4  0x00007fffefad2112 in std::__glibcxx_assert_fail(char const*, int, char const*, char const*) (file=<optimized out>, line=<optimized out>, function=<optimized out>, condition=<optimized out>)
    at /usr/src/debug/gcc/libstdc++-v3/src/c++11/debug.cc:60
#5  0x00007fff7766ca95 in  () at /opt/rocm/lib/libhsa-runtime64.so.1
#6  0x00007fff7766cf08 in  () at /opt/rocm/lib/libhsa-runtime64.so.1
#7  0x00007fff7766d09c in  () at /opt/rocm/lib/libhsa-runtime64.so.1
#8  0x00007fff7764d47f in  () at /opt/rocm/lib/libhsa-runtime64.so.1
#9  0x00007fffa80b0b6b in  () at /opt/rocm/hip/lib/libamdhip64.so.5
#10 0x00007fffa7eaa108 in  () at /opt/rocm/hip/lib/libamdhip64.so.5
#11 0x00007fffa7ecafad in hipGetDeviceCount () at /opt/rocm/hip/lib/libamdhip64.so.5
#12 0x00007fffef9e8057 in c10::hip::device_count() () at /usr/lib/python3.10/site-packages/torch/lib/libc10_hip.so
#13 0x00007fffef3e88b2 in THCPModule_getDeviceCount_wrap(_object*, _object*) () at /usr/lib/python3.10/site-packages/torch/lib/libtorch_python.so

libhsa-runtime64.so seems to come from the hip-runtime-amd package. Could it have been built with some kind of debug assertions enabled?

Related: arch4edu/arch4edu#203

[jellersby]

@tpkessler Are the libs being built with -D_GLIBCXX_ASSERTIONS?

[tpkessler]

Hi @jellersby! Yes, all packages are built with glibc assertions enabled (the default on Arch Linux). I've patched the package in [community-testing]. Could you please try to run it again?

[Funestia]

I had the same issue and it seems to be fixed now.

[jellersby]

Thanks @tpkessler, that worked for me too. Out of curiosity, did you remove the assertions or did you patch the code?

[tpkessler]

I patched the code. Assertions are still enabled.

[tpkessler]

The full ROCm stack is now in [community], including PyTorch with ROCm support, see the official announcement on the mailing list:

https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/thread/NKV3ETKNL56NEUKT53GKRTHUANAKMHMV/

[wuxxin]

thanks for your work, pyTorch on gfx1030 is working for me,
i updated the build of torchvision for rocm https://github.com/rocm-arch/python-torchvision-rocm
to the corresponding pyTorch package, currently in AUR

[tpkessler]

Awesome! I'm very happy to hear that.

[sasha0552]

Can you add gfx803 to the list of PyTorch architectures? The ROCm stack can already work with gfx803, but python-pytorch-rocm cannot.
I built and tested python-pytorch-rocm with gfx803, it works.
Just add gfx803 to PYTORCH_ROCM_ARCH.

[tpkessler]

Hi! I've updated the package to include gfx803 and pushed it to [community-testing]. Could you please tell me if it works?

[sasha0552]

It works! Thanks

$ python
Python 3.10.9 (main, Dec 19 2022, 17:35:49) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.get_arch_list()
['gfx803', 'gfx900', 'gfx906', 'gfx908', 'gfx90a', 'gfx1030']
>>> torch.cuda.is_available()
True
>>> torch.cuda.get_device_name(0)
'AMD Radeon RX Graphics'
>>> d = torch.device("cuda")
>>> a = torch.rand(1, 2).to(d)
>>> print(a + 0)
tensor([[0.2274, 0.5280]], device='cuda:0')

[wuxxin]

i added gfx803 to the torchvision-rocm build, to mirror the arch list of pytorch-rocm