CUDA error: forward compatibility was attempted on non supported HW

[ayan4m1]

I am getting an error running in Docker using the docker-compose.yml from the documentation site after upgrading to this nvidia driver in Ubuntu:

CUDA error: forward compatibility was attempted on non supported HW 

ii  nvidia-driver-550                              550.120-0ubuntu0.24.04.1                    amd64        NVIDIA driver metapackage

[giladgd]

Many CUDA errors happen due to mismatching versions of the drivers and the rest of the CUDA tools, so try to upgrade everything CUDA-related to the latest version to ensure all the versions match.
Also, it's important for the CUDA version used in the docker image to match or be older than the CUDA version installed on the host machine, so consider downgrading it if you need.

[ayan4m1]

Yeah, unfortunately I'm caught between two pieces of software - a game that needs the latest version of nvidia-drivers, and node-llama-cpp needing older ones. At least it doesn't seem to stop inferencing outright. Thanks.

[giladgd]

What's the version of CUDA installed on your machine?
node-llama-cpp prebuilt binaries are compiled with CUDA 12.2 to ensure all newer CUDA drivers can use them.

[ayan4m1]

12.6, but I'm manually building CUDA binaries in my Dockerfile according to the docs, so I'm not using prebuilt binaries afaik. How would I rule it out?

[giladgd]

Do you build the binaries outside the docker image or inside of it?
The prebuilt CUDA binaries have better compatibility with various hardware and machine configurations than binaries you can build from source on your machine, thus it's recommended to use them if possible.
I'll release a new version that significantly improves the performance of the prebuilt CUDA binaries in the next week or so, so you shouldn't notice a performance difference between the prebuilt binaries and a local build anymore after that release, if that's the reason you build from source.

[ayan4m1]

Inside the docker image.

Gotcha - the reason I build from source is just that I assumed it was necessary for CUDA (e.g. non-CPU). I can try relying on prebuilt binaries but I don't know how to ensure CUDA ones will be selected. I will go back to the docs.