More up-to-date documentation on CUDA backend

[danchitnis]

I have been following the documentation in this repository to compile SYCL code for CUDA. I found few discrepancies which I would appreciate if someone could clarify:

1. The -fsycl-targets=nvptx64-nvidia-cuda-sycldevice doesn't seem to make any difference when compiling for CUDA. It always detects CUDA backend with or without this flag
2. The LD_LIBRARY_PATH needs to be set for runtime. Although this is mentioned in the FAQ, it would be better if it is mentioned during the starting guide, otherwise, it is very difficult to figure out why this is needed after the compilation.
3. when I compile for CUDA then my Intel CPU is not detected as a backend. However, when I use the original OneAPI installation, it detects my Intel CPU, but of course, not CUDA. Is there any way to get these detected at the same time? What is the environment variables to direct to Intel's CPU backend? I believe this is the most popular configuration.

[alexbatashev]

The -fsycl-targets=nvptx64-nvidia-cuda-sycldevice doesn't seem to make any difference when compiling for CUDA. It always detects CUDA backend with or without this flag

Do you mean in runtime? Compiler flags have almost zero effect on what runtime does. If it finds CUDA GPU to be the only available device, it will use it no matter what. You can check the available devices with sycl-ls tool that comes with both oneAPI base kit and open-source installations.

The LD_LIBRARY_PATH needs to be set for runtime. Although this is mentioned in the FAQ, it would be better if it is mentioned during the starting guide, otherwise, it is very difficult to figure out why this is needed after the compilation.

Not sure how come so, that documentation is missing this information, but here's a short guide how to install open-source compiler right into your system:

1. To configure.py add the following argument: --cmake-opt="-DCMAKE_INSTALL_PREFIX=/path/to/install/dir". If you don't do this, /usr/local will be used.
2. Run ninja deploy-sycl-toolchain target. You may want to build sycl-toolchain target first, and then run deploy under root.
3. If you've installed your toolchain to a non-default location, you need to specify PATH and LD_LIBRARY_PATH (e.g. in your .bashrc). You can also use ldconfig to teach your system look for libsycl.so.

when I compile for CUDA then my Intel CPU is not detected as a backend. However, when I use the original OneAPI installation, it detects my Intel CPU, but of course, not CUDA. Is there any way to get these detected at the same time? What is the environment variables to direct to Intel's CPU backend? I believe this is the most popular configuration.

Did you follow this section of GSG? oneAPI's setenv.sh does this automatically, but with open-source compiler you need to follow the procedure manually.

[alexbatashev]

@danchitnis I wouldn't expect build to require a real GPU. Would you please share the kind of error you're getting?

[danchitnis]

@alexbatashev I run the build container 5 times without GPU access and one time with GPU access (--gpus all). Here are the log files:

Failed: log1, log2, log3, log4, log5

Success: log6

You can see that the build fails the following lines:

[  6%] Building CXX object tools/sycl/source/CMakeFiles/sycl_object.dir/spirv_ops.cpp.o
[  6%] Building CXX object tools/sycl/source/CMakeFiles/sycl_object.dir/detail/posix_pi.cpp.o
[  6%] Built target sycl_object
make: *** [Makefile:152: all] Error 2

or

[  6%] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/WebAssemblyDisassemblerEmitter.cpp.o
[  6%] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CTagsEmitter.cpp.o
[  6%] Built target obj.llvm-tblgen
make: *** [Makefile:152: all] Error 2

I did a separate test to see if nvcc compiles without GPU access, and indeed it does compile successfully. However, when I try to run, it generates the following error which is expected (e.g. for deviceQuery sample):

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

I assume somewhere in your llvm build code, you are looking for the CUDA version, hence it fails?

[alexbatashev]

@danchitnis that's not the error you're looking for. What matters is this piece from the beginning of log files:

[  1%] Built target scan-build-py
Scanning dependencies of target sycl_object
[  1%] Building CXX object tools/sycl/source/CMakeFiles/sycl_object.dir/backend/opencl.cpp.o
[  1%] Building CXX object tools/clang/utils/TableGen/CMakeFiles/obj.clang-tblgen.dir/ClangASTNodesEmitter.cpp.o
[  1%] Linking CXX static library ../../../lib/libxpti.a
[  1%] Built target xpti
Scanning dependencies of target pi_cuda
make[2]: *** No rule to make target 'CUDA_CUDA_LIBRARY-NOTFOUND', needed by 'lib/libpi_cuda.so'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[  1%] Building CXX object tools/sycl/plugins/cuda/CMakeFiles/pi_cuda.dir/pi_cuda.cpp.o

It's kinda out fault that we don't check this in configuration time. It looks like CMake was able to find nvcc, but failed to locate runtime libraries. As a solution, you can try setting CUDA_PATH and CUDA_LIB_PATH env variables.

[danchitnis]

@alexbatashev thanks for finding the error message. It seems this was a known bug within Docker/CUDA/CMAKE. I fixed this with CMAKE_LIBRARY_PATH=/usr/local/cuda-11.4/lib64/stubs. I am now able to to build the docker images in the cloud, however, your recent commit broke it again! The error log is here:

#10 122.5 [  7%] Building CXX object tools/sycl/tools/pi-trace/CMakeFiles/pi_trace.dir/pi_trace.cpp.o
#10 122.7 In file included from /opt/build/llvm/sycl/include/CL/sycl/detail/common.hpp:11,
#10 122.7                  from /opt/build/llvm/sycl/include/CL/sycl/detail/pi.hpp:17,
#10 122.7                  from /opt/build/llvm/sycl/tools/pi-trace/../xpti_helpers/pi_arguments_handler.hpp:11,
#10 122.7                  from /opt/build/llvm/sycl/tools/pi-trace/pi_trace.cpp:15:
#10 122.7 /opt/build/llvm/sycl/include/CL/sycl/detail/cl.h:17:10: fatal error: CL/cl.h: No such file or directory
#10 122.7    17 | #include <CL/cl.h>
#10 122.7       |          ^~~~~~~~~
#10 122.7 compilation terminated.
#10 122.7 make[2]: *** [tools/sycl/tools/pi-trace/CMakeFiles/pi_trace.dir/build.make:63: tools/sycl/tools/pi-trace/CMakeFiles/pi_trace.dir/pi_trace.cpp.o] Error 1
#10 122.7 make[1]: *** [CMakeFiles/Makefile2:81982: tools/sycl/tools/pi-trace/CMakeFiles/pi_trace.dir/all] Error 2
#10 122.7 make[1]: *** Waiting for unfinished jobs....

Should I create an issue for this?

[bader]

I think this one is already reported as #4137.